Skip to content

PERF: Compute whole-array statistics and copy OOC stores on all cores - #1924

Open
joeykleingers wants to merge 13 commits into
BlueQuartzSoftware:developfrom
joeykleingers:ooc-bulk-io-performance
Open

joeykleingers wants to merge 13 commits into
BlueQuartzSoftware:developfrom
joeykleingers:ooc-bulk-io-performance

Conversation

@joeykleingers

@joeykleingers joeykleingers commented Nov 5, 2026 •

Copy link
Copy Markdown
Contributor

Summary

Whole-array statistics in Compute Attribute Array Statistics use all cores with bounded memory, and store copies into an out-of-core store write whole chunks in pieces of up to 128 MiB. A store copy no longer zero-fills a destination that the copy then replaces. A test HDF5 file driver whose close fails is added for out-of-core store tests. 18 files changed, +3934 / −342 lines.

Dependencies: none. The branch is rebased on develop at 010b34e (#1915). BlueQuartzSoftware/SimplnxOoc#20 used the chunk-aligned copy and the test file driver, but it is now a draft and must be redesigned on the out-of-core merge train (#22–#32). This PR does not depend on it. No code on develop uses FailCloseVfd yet.

1. Whole-array statistics on all cores

  • src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/ComputeArrayStatistics.{hpp,cpp} — the histogram engine, the two-pass engine and the chunk-aligned standardization.
  • src/Plugins/SimplnxCore/test/ComputeArrayStatisticsTest.cpp — oracle, determinism, Boolean and cancellation tests.

Boolean, 8-bit and 16-bit integer input uses one pass into per-thread value histograms. Length, extrema, sum, mean, median, mode and unique count are exact. A block histogram of at most 256 counters is counted in a stack-local array and added to the thread histogram once for each block.

Other types without a median, mode or unique-count request use two passes. Each pass reduces fixed blocks in parallel and merges them in block order, so the result does not depend on the thread count or on the page size. The mean and deviation formulas do not change. Pages are 32 MiB over the value and mask buffers and are read serially on the calling thread.

Standardization reads and writes pieces that follow the chunk grid of its output and transforms each piece in parallel. Without a mask, it does not read the current standardized values.

Boolean pages are made canonical after each bulk read, so a stored byte such as 255 counts as one true value. Without this, such a byte was used as a histogram index past the end of the two-counter histogram.

A cancellation in the deviation pass returns before the standard deviation is computed from partial sums.

Grouped statistics, and wide types with a median, mode or unique-count request, keep the existing implementation. Test-only page, block and witness seams exist only when SIMPLNX_BUILD_TESTS is set.

2. Chunk-aligned store copies

  • src/simplnx/Utilities/ChunkAlignedTransfer.{hpp,cpp} — plans flat tuple ranges as pieces that are aligned or unaligned against a tuple-rank chunk grid.
  • src/simplnx/DataStructure/AbstractDataStore.hpp — copyFrom() routing.
  • src/simplnx/Utilities/StoreCopyUtilities.{hpp,cpp} — creates a nonempty copy destination with DataStoreInitializationMode::DeferredZeroFill.

AbstractDataStore::copyFrom() copies into an out-of-core store in chunk-aligned pieces of at most k_AlignedCopyTargetBytes (128 MiB), when the copy is larger than 1 MiB, the destination reports a chunk shape and the copy holds at least one aligned unit. The grid is queried again before each piece, because the first write to a read-only destination can replace its grid. All other copies use the unchanged 1 MiB loop. The documentation of the store copy functions states the 128 MiB buffer of an aligned copy.

A store copy creates its destination with deferred zero-fill, because the copy replaces every value and a failed copy discards the destination. An in-memory factory ignores the mode.

3. Test HDF5 file driver

  • test/UnitTestCommon/include/simplnx/UnitTest/FailCloseVfd.hpp — an HDF5 virtual file driver that writes normally and reports a failure from its close callback when a test arms it.

4. Coding knowledge base

  • docs/Coding_Knowledge_Base.md — new entry Charge large scratch buffers to the working-memory budget: a scratch buffer whose size grows with a chunk, page, piece or the thread count reserves its bytes with reserveWorkingMemory() and is sized from the grant.

Test plan

Results on head 8d4e537 (develop 263031b + 12 commits). The later rebase onto 010b34e had no conflicts in code files and added only a documentation commit; it was not rebuilt locally:

  • macOS simplnx-only build (ci-macos-arm64, Release): the full build succeeds with no warnings in the changed files.
  • macOS simplnx-only ctest (serial): 3995 of 3995 tests pass, including the ComputeArrayStatistics, ChunkAlignedTransfer and DataStore copyFrom tests.
  • macOS OOC-Release (DREAM3D-NX with DREAM3DNX_ENABLE_OOC=ON): not yet re-run after the rebase.
  • Windows and Linux CI pass.

Results before the rebase (old base a7dffc4), reported by the author:

  • macOS OOC-Release: full serial ctest passed 4598 of 4603 tests, including 28 ComputeArrayStatistics and 3 ChunkAlignedTransfer tests. The 5 failures also failed on develop at that time.
  • macOS in-core build: full serial ctest passed 3868 of 3869 tests. The one failure, SimplnxCore::CropImageGeometry: real OOC stores preserve cropped values, needs an out-of-core storage format that an in-core build does not register.

Related Issue

  • This PR has no related issue

@joeykleingers joeykleingers self-assigned this Nov 5, 2026
@imikejackson
imikejackson force-pushed the ooc-bulk-io-performance branch from 923fbe9 to 8d4e537 Compare November 13, 2026 19:18

@imikejackson imikejackson left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

PR 1924 Code Review

Review of commit 8d4e53714, the branch after a maintainer rebased it onto develop 263031b. Coverage: all 17 changed files.

On the rebased head, macOS ci-macos-arm64 Release builds with no warnings in the changed files. A full serial ctest run passes 3995 of 3995 tests.

Findings

  • [Important] The 128 MiB aligned copy buffer is not charged to the working-memory budget
    File: src/simplnx/DataStructure/AbstractDataStore.hpp:1408-1412 (copyFromChunkAligned)
    An aligned copy now allocates up to 128 MiB, where the old loop allocated 1 MiB. CacheMemoryBudgetManager::reserveWorkingMemory() does not count it. Parallel callers multiply it. RegularizeZSpacing.cpp:51 calls copyFrom() from ParallelTaskAlgorithm tasks. With 8192² float32 slices into out-of-core stores on 16 tasks, peak scratch goes from about 16 MiB to about 2 GiB, all outside the budget.
    Fix: reserve working memory for each piece and size the piece from the grant, with a 1 MiB minimum. Or document that a copy owns up to 128 MiB that is not counted, and limit the parallel callers.

  • [Minor] The whole-array statistics buffers are not charged to the budget
    File: ComputeArrayStatistics.cpp:2027, :2385, :2848
    32 MiB of value and mask pages. For 16-bit input, add 512 KiB of histogram for each TBB thread plus one merged copy: about 64 MiB with 128 threads. ComputeFeatureCentroids.cpp:266 already charges its buffers.
    Fix: reserve these bytes, and shrink the page to the grant, but never below one block.

  • [Minor] The aligned path is used when one aligned unit is larger than a piece
    File: AbstractDataStore.hpp:1112
    The routing checks totalSrcTuples >= unitTuples, but not unitTuples <= maxPieceTuples. For tuple shape {2048, 2048, 2048} and chunk shape {64, 64, 64}, one unit is 1 GiB of float32. Every piece is then an unaligned 128 MiB write: 128 times the memory, and no aligned writes.
    Fix: also require unitTuples <= maxPieceTuples. Otherwise use the 1 MiB loop.

  • [Minor] FailCloseVfd.hpp is not used and is not compiled
    File: test/UnitTestCommon/include/simplnx/UnitTest/FailCloseVfd.hpp, test/UnitTestCommon/CMakeLists.txt:31
    No code in simplnx develop or in the companion repository's develop includes it. It is a header listed as an INTERFACE source, so no target compiles it, and it can break without notice. Other problems:

    • The driver value 4243 is outside the range 256–511 that H5FDpublic.h reserves for testing drivers.
    • The return values of fclose, ferror after a short fread, Seek and H5Pset_driver are ignored. A driver that tests close failures should check fclose.
    • Plausible: the cached static driver identifier becomes invalid after H5close()/H5open(). Arming the driver then does nothing.
      Fix: remove it from this PR, and add it with the first PR that uses it.
  • [Minor] The aligned copy is tested only with a store that pretends to be out-of-core
    File: test/DataStoreBulkTest.cpp:701-880
    ChunkReportingDataStore is a resident DataStore<int32> that only reports StoreType::OutOfCore. No test uses a disk-backed store. These cases are not tested:

    • a nonzero srcTupleOffset on the aligned path; all aligned cases use source offset 0, so an offset applied twice would still pass;
    • a read or write failure in the middle of the aligned loop;
    • T = bool;
    • totalSrcTuples == unitTuples;
    • a copy just over 1 MiB.

    The DeferredZeroFill change has no test, because the in-memory factory ignores the mode.
    Fix: add these cases, plus one out-of-core store contract test that copies with CopyDataStore() and checks that the backend received aligned writes.

  • [Minor] The determinism test runs almost nothing in parallel for 8-byte types
    File: ComputeArrayStatisticsTest.cpp:2267, ComputeArrayStatistics.hpp:156
    With a 1024-byte page, 64-tuple blocks and a Boolean mask, a uint64 or float64 page holds one block. The first serial versus parallel comparison then runs nothing in parallel for those types. The witness counts calls to runPageBlocks, not concurrency, so the Doxygen claim that it proves "its parallel path ran" is too strong.
    Fix: choose page sizes for each type that give at least 4 blocks for each page, and reword the witness Doxygen.

  • [Minor] A cancelled engine returns a valid result with partial statistics
    File: ComputeArrayStatistics.cpp:2545, :2726
    Only the caller check at :2947 stops the partial values from being written. The new cancellation test at ComputeArrayStatisticsTest.cpp:2452 also passes without the deviation-pass fix.
    Fix: return a distinct cancelled state, for example Result<std::optional<WholeArrayStatistics<T>>>, so a later caller cannot write partial values.

  • [Minor] No progress messages in the whole-array engines
    The engines send no progress during up to three passes (moments, deviation, standardize). Out of core, these passes can take a long time.
    Fix: one ThrottledMessageHandler that the calling thread updates once for each page. Workers never touch it.

  • [Minor] IsChunkAlignedTupleRange() documentation
    File: src/simplnx/Utilities/ChunkAlignedTransfer.hpp:36-62

    • The comment describes the acceptance rule of a writer in another repository and says "Keep both rules identical". Nothing enforces this. If the rules drift, pieces still copy correctly but stop using the fast path. Describe the rule in general words, and have the other writer call this public function.
    • "covers only whole chunks" is a sufficient condition, not a necessary one. For tuple shape {4, 4}, chunk shape {1, 2}, start 2 and count 4, the range covers only whole chunks, but the function returns false. Suggested text: "Tells if the range has the shape that a chunk-granular writer accepts. This is a sufficient condition for whole-chunk coverage, not a necessary one."
  • [Nit] Test assertions

    • DataStoreBulkTest.cpp:797, 806, 814, 823, 833, 845, 861 use REQUIRE(result.valid()). Use SIMPLNX_RESULT_REQUIRE_VALID(result), so a failure prints the errors.
    • The new ComputeArrayStatisticsTest.cpp cases do not wrap getDataRefAs in REQUIRE_NOTHROW and do not call UnitTest::CheckArraysInheritTupleDims(dataStructure).
    • The out-of-core test at ComputeArrayStatisticsTest.cpp:2297 asserts that the input and mask are disk-backed, but not the Standardized output. That output is the array that uses the chunk-aligned write path.
  • [Nit] Naming against .clang-tidy

    • g_ScanPageBytesOverride and the other g_ globals in ComputeArrayStatistics.cpp, and g_AlignedCopyTargetBytesOverride in ChunkAlignedTransfer.cpp:14, break GlobalVariableCase: CamelCase.
    • Public members ChunkAlignedPiece::startTuple/tupleCount/isChunkAligned and GridGeometry::chunkExtents/strides break PublicMemberCase: CamelCase.
    • Tests: RecordedWrite::startIndex/valueCount, public m_ members of ChunkReportingDataStore, and the Pieces alias (no Type suffix).
    • FailCloseVfd: s_Armed, s_CloseFailureCount, members pub/fp/eoa/eof, and pointers without the Ptr suffix.
  • [Nit] Comments and contracts

    • AbstractDataStore.hpp:1411: make_unique_for_overwrite<T[]> is allowed because T can be bool. State that reason next to the declaration, as the project rule for unique_ptr<T[]> asks.
    • copyFrom() Doxygen (AbstractDataStore.hpp:1050-1071): state that pieces already written stay written after an error, and that copies between overlapping ranges of one store are not supported.
    • IDataStore::getChunkShape() (IDataStore.hpp:118-123): state that the shape has tuple rank. A backend that returns tuple plus component rank silently gets the 1 MiB loop.
  • [Nit] Simplifications

    • ComputeArrayStatistics.cpp:1950: an array of eight std::function objects is built only to run the writes in order. Eight if blocks, or a small helper that returns early on cancel, are simpler and do not allocate.
    • Rename TodayScanlineOracle (ComputeArrayStatisticsTest.cpp:1625) to a name that will not become stale, for example ElementOrderReference.
    • PlanChunkAlignedPieces() is exported, but only tests use it.

Pre-existing defects (not introduced by this PR)

These produce wrong data:

  • All-NaN float input. Float input of only NaN gives Minimum FLT_MAX and Maximum -FLT_MAX with no warning. develop does the same. The new test at ComputeArrayStatisticsTest.cpp:2168 asserts these sentinel values, so they become a tested contract. Please do not assert them as correct behavior. Mark the case as a known defect (related: #1867, #1874).
  • Ignored copyFrom() results. RegularizeZSpacing.cpp:51, PadImageGeometry.cpp:65 and ConvertOrientationsToVertexGeometry.cpp:35 ignore the result. A failed copy leaves zeros with no error. copyFrom() is not [[nodiscard]].

These do not produce wrong data:

  • The filter documentation disagrees with the code (docs/ComputeArrayStatisticsFilter.md:15, 30, 32). The new tests now assert the real behavior, so this is a good time to correct the page:
    • Boolean mean and standard deviation are documented as "true if there are more instances of true". The code computes trueCount / length.
    • "If Standardize Data is chosen for a boolean array, no modifications are made". The code writes (b - mean) / std.
    • Boolean Mode is documented as "computed as normal". The code returns error -57305.
    • "The standardized array is computed in double precision". It is computed in 32-bit float.
    • Length is documented as "signed 64-bit integer". The output is a UInt64Array.
  • The range checks in copyFrom() (errors -14602 and -14603) multiply without an overflow check.

Missing evidence

  • clang-tidy was not run. clang-format 19.1.1 --dry-run --Werror passes on all changed C++ files.
  • The DREAM3D-NX out-of-core build was not re-run on the rebased head. The only tests with real disk-backed stores are the out-of-core statistics tests, and they ran only in the author's run before the rebase.
  • Windows and Linux CI are still pending on the new head.
  • No performance numbers, in-core or out-of-core, and no memory measurement with concurrent copies.

Confirmed correct

  • Histogram engine: histogramBin offsets by numeric_limits<T>::min(), so bins are in value order for int8 and int16. The wrapping uint64 sum is exact. Median positions (n-1)/2 and n/2, mode order and unique count match the sorted path. Empty, fully masked and one-tuple inputs give the develop outputs.
  • Two-pass engine: pages are whole multiples of the block, so the block grid is global. The merge in block order on the calling thread does not depend on thread count or page size. Strict comparisons against the max()/lowest() sentinels keep the develop handling of NaN and -0/+0.
  • Routing: the engines run only when !ComputeByIndex && GroupCount == 1. All other requests use the unchanged path. Boolean with Mode still fails with -57305 before routing.
  • Threads: every copyIntoBuffer()/copyFromBuffer() call is on the calling thread. Workers use only buffers that the caller owns, write to separate slots, and use enumerable_thread_specific histograms. No worker sends messages.
  • Boolean pages are made canonical after every read, on every path.
  • The chunk planner was compiled alone and checked against an independent whole-chunk-coverage oracle on 51,202 start positions (rank 1–3, dimensions 1–5, chunk extents 1–6, random caps). Aligned pieces never cover a partial chunk, and pieces are contiguous, capped and not empty.
  • Asking for the grid again before each piece means a stale grid after promotion costs only speed, not correctness.
  • CopyDataStore() is the only path that uses DeferredZeroFill, and a failed copy discards its destination. The growth initializer from develop is set after the copy.
  • SIMPLNX_BUILD_TESTS is a PUBLIC compile definition of simplnx, so the plugin, the library and the tests see the same declarations.

Verdict

REQUEST CHANGES. The new statistics engines and the copy routing are correct. The blocker is the 128 MiB copy buffer, which the working-memory budget does not count, and which parallel callers multiply. Please also remove FailCloseVfd.hpp until a PR uses it.

joeykleingers and others added 13 commits November 13, 2026 15:35
* CopyNumericFunctor creates a nonempty destination with
  DataStoreInitializationMode::DeferredZeroFill. The copyFrom() that
  follows replaces every value, and a failed copy discards the
  destination, so the initial zero-fill only wrote each value twice.
  An in-memory factory ignores the mode.
* Add ChunkAlignedTransfer (IsChunkAlignedTupleRange,
  ComputeNominalChunkAlignedUnitTuples, ComputeNextChunkAlignedPiece,
  PlanChunkAlignedPieces). It splits a flat tuple range into pieces
  that are marked aligned or unaligned against a tuple-rank chunk grid.
  The alignment rule is the rule that the out-of-core parallel chunk
  writer uses to accept a tuple write. No piece is larger than the
  cap, also when one aligned unit is larger than the cap.
* AbstractDataStore::copyFrom() plans pieces of at most 32 MiB against
  the destination chunk grid when the copy is larger than 1 MiB, the
  destination is out of core, the destination reports a chunk shape,
  and the copy holds at least one nominal aligned unit. The grid is
  queried again before each piece, because the first write to a
  read-only destination can replace its grid. All other copies use the
  unchanged 1 MiB loop.
* Add ChunkAlignedTransferTest and a DataStoreBulkTest case for the
  copyFrom() routing, including a grid that changes after the first
  write.

Signed-off-by: Joey Kleingers <joey.kleingers@bluequartz.net>
* copyFromChunkAligned() allocates its piece buffer with
  std::make_unique_for_overwrite. std::make_unique value-initialized up
  to 32 MiB for each call. copyIntoBuffer() overwrites each element of
  a piece before copyFromBuffer() reads it.
* Document ChunkAlignedPiece::startTuple and
  ChunkAlignedPiece::tupleCount.

Signed-off-by: Joey Kleingers <joey.kleingers@bluequartz.net>
The bounded whole-array scan in ComputeArrayStatistics read 64 Ki tuple
pages and reduced every value on the calling thread. Two full passes of
a 43.8 GB uint16 array used about 2.5 cores.

* Boolean, Int8, UInt8, Int16, and UInt16 input now uses one pass into
  per-thread value histograms. Length, extrema, sum, mean, median, mode,
  and unique count are exact. The deviation uses the same per-value
  expression, so only its Float64 summation order changes.
* Other types without a median, mode, or unique-count request use two
  passes. Each pass reduces fixed blocks into slots in parallel and
  merges the slots in block order, so the result does not depend on the
  thread count or the page size. The mean and deviation formulas do not
  change.
* Pages are 32 MiB over the value and mask buffers and are read
  serially. Blocks lie on a global tuple grid.
* Standardization reads and writes pieces that follow the chunk grid of
  the standardized store and transforms each piece in parallel. Without
  a mask, it does not read the current standardized values.
* Grouped statistics, and wide types with a median, mode, or unique
  count request, keep the existing implementation.
* Exported page, block, and witness seams let tests use small arrays.
* The resident per-feature tbb::parallel_for has a serial fallback when
  SIMPLNX_ENABLE_MULTICORE is off.

Signed-off-by: Joey Kleingers <joey.kleingers@bluequartz.net>
* Compare every histogram type and every two-pass type with an
  independent oracle of the element-order scanline formulas. The cases
  cover one tuple, partial pages, many pages, Boolean, UInt8, and
  all-masked masks. Integer results and histogram order statistics must
  be exact. The deviation must agree within 1e-6 relative. Float sums
  must agree within n * eps * sum(|x|).
* Cover ties, several modes, even and odd lengths, Boolean medians,
  and negative Int8 and Int16 values.
* Check NaN, signed-zero, and infinity semantics of the two-pass
  engine, and parity with the Direct path on finite data.
* Require bit-identical results with one TBB thread and with the
  default limit, and with two page sizes.
* Require the parallel dispatch witness on genuine out-of-core stores,
  and prove that wide types with order statistics keep the sorted path.
* Check that a cancellation in the deviation pass leaves the outputs
  unwritten.

Signed-off-by: Joey Kleingers <joey.kleingers@bluequartz.net>
* AbstractDataStore::copyFrom() now reads and writes a chunk-aligned
  copy into an out-of-core store in pieces of at most
  k_AlignedCopyTargetBytes (128 MiB). The previous target was 32 MiB.
* A piece can now hold more than 64 of the default 1 MiB session
  chunks. The parallel deflate writer can then compress one batch
  while it commits the previous batch. Larger pieces also need four
  times fewer source-read and destination-write handoffs.
* SetAlignedCopyTargetBytesForTesting() and
  GetAlignedCopyTargetBytesForTesting() let a test build change the
  piece target. They are exported and have their storage in
  ChunkAlignedTransfer.cpp, so every module shares one value.
* The copy documentation now states the 128 MiB buffer of an aligned
  copy. Several deepCopy documents stated a 1 MiB limit that did not
  apply to aligned copies.

Signed-off-by: Joey Kleingers <joey.kleingers@bluequartz.net>
* A copyFrom() routing section sets a three-slice piece target and
  checks that a twelve-slice copy writes four aligned pieces.
* A new case checks the 128 MiB production target and that a target
  of 0 restores it.

Signed-off-by: Joey Kleingers <joey.kleingers@bluequartz.net>
* A store can move bool values as raw bytes. A .dream3d Boolean dataset
  from h5py, MATLAB, or a legacy writer can then supply bytes such as
  255. In an optimized build histogramBin<bool> used such a byte as a
  counter index past the two-counter histogram, which is a heap write.
  A Boolean mask with such a byte added a selection weight of up to 255
  in the histogram engine. The other engines skipped tuples whose mask
  byte was nonzero but not 1.
* readStatisticsTuples() fills every value and mask page of the
  scanline statistics. It now rewrites each byte of a bool page to 0 or
  1 through unsigned char after a successful read. The histogram,
  two-pass, Standardize, and sorted exact paths then count every
  nonzero byte as one true value.
* Add tests that write the bytes {1, 2, 127, 128, 255} for true values
  into Boolean input and mask stores, in memory and in genuine
  out-of-core stores. The tests require the bytes to reach the store
  unchanged, then compare every statistic with the canonical oracle.

Signed-off-by: Joey Kleingers <joey.kleingers@bluequartz.net>
* computeMomentStatistics() now checks shouldCancel after the deviation
  pass, as it does after the moment pass. A cancelled pass returns
  before the standard deviation is computed from partial sums, so no
  caller can publish that value.
* generateWholeArrayStatistics() already checks shouldCancel after the
  engine returns, so the filter output does not change.

Signed-off-by: Joey Kleingers <joey.kleingers@bluequartz.net>
* The per-thread histograms of Boolean and 8-bit input are 16 bytes to
  2 KiB. Histograms of different threads can then share a cache line,
  and each increment can invalidate that line for another thread.
* countBlockValues() now counts a block into a stack-local
  std::array when the histogram has at most 256 counters, and adds it
  to the thread's counters once for each block. 16-bit input keeps
  counting into its 512 KiB thread histogram.
* Integer counts commute, so the statistics do not change.

Signed-off-by: Joey Kleingers <joey.kleingers@bluequartz.net>
* The filter creates a zero-filled standardized output. A piece buffer
  that holds zeros could then hide a missing read-back of the current
  values before a masked Standardize piece is written.
* The new test runs the algorithm directly on an output filled with
  42.0F. It uses Boolean and UInt8 masks on the histogram engine
  (uint16) and the two-pass engine (int32, float32) over several pieces.
  Unselected tuples must keep 42.0F, and selected tuples must equal
  (value - mean) / deviation with the reported mean and deviation.

Signed-off-by: Joey Kleingers <joey.kleingers@bluequartz.net>
* The comments on k_AlignedCopyTargetBytes and copyFromChunkAligned()
  named an out-of-core chunk size. They now refer to destination
  chunks, so the core headers do not name an out-of-core concept.
* Both comments name HDF5::ParallelChunkCodec::deflateSpanIntoChunks()
  and k_MaxPreparedBatchChunks, so a search finds the coupling between
  the piece size and the codec batch limit.

Signed-off-by: Joey Kleingers <joey.kleingers@bluequartz.net>
* FailCloseVfd is a test-only HDF5 virtual file driver. It writes
  normally and reports a failure from its close callback when a test
  arms it.
* The out-of-core store tests use it to prove that a copy-on-write
  promotion releases its replacement file when the close of that file
  fails.

Signed-off-by: Joey Kleingers <joey.kleingers@bluequartz.net>
Signed-off-by: Michael Jackson <mike.jackson@bluequartz.net>
Add a Coding Knowledge Base entry: a scratch buffer whose size grows with a
chunk, page, piece or the thread count reserves its bytes with
reserveWorkingMemory() and is sized from the grant. The review of this PR
found the rule broken in the chunk-aligned copyFrom() buffer and in the
whole-array statistics pages.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@imikejackson
imikejackson force-pushed the ooc-bulk-io-performance branch from ca3f1d4 to 726076c Compare November 13, 2026 20:35
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants