Skip to content

BUG: Fix 21 filter defects ported from 'SIMPL' and 'DREAM3D' - #1915

Merged
imikejackson merged 33 commits into
BlueQuartzSoftware:developfrom
imikejackson:topic/errata_1908_fixes
Nov 13, 2026
Merged

imikejackson merged 33 commits into
BlueQuartzSoftware:developfrom
imikejackson:topic/errata_1908_fixes

Conversation

@imikejackson

@imikejackson imikejackson commented Nov 4, 2026 •

Copy link
Copy Markdown
Contributor

Summary

Fixes 21 filter defects from simplnx#1908 that DREAM3D-NX shares with DREAM.3D 6.5.171. Each one is listed on the 6.5.171 Known Issues page (KI-007 to KI-083). Each BUG: commit fixes one #1908 item, or two items with the same cause, and starts with a unit test that fails on develop. Expected values come from hand calculations or closed forms, not from filter output. Where the fixed output differs from 6.5.171, the filter documentation says so. 82 files changed, +2343 / −170 lines.

The REV: commits and the last DOC: commit answer the reviews. They make each fix surgical: code that went beyond the defect filed in #1908 is removed, and the review findings in the remaining code are fixed.

Every item was reviewed in alternating rounds by two independent reviewers until two consecutive rounds found nothing.

1. Wrong output (WRONG_DATA)

  • src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/AlignSectionsFeatureCentroid.cpp, src/simplnx/Utilities/AlignSections.cpp (L04)
  • .../ComputeFeatureClustering.cpp (L11)
  • .../ComputeVectorColors.cpp (L12)
  • .../QuickSurfaceMeshDirect.cpp, .../QuickSurfaceMeshScanline.cpp (L23)
  • src/simplnx/Utilities/ImageRotationUtilities.cpp (L28)
  • .../WriteVtkRectilinearGrid.cpp (L29)
  • src/Plugins/OrientationAnalysis/src/OrientationAnalysis/utilities/IEbsdOemReader.hpp (L54)
  • .../WriteINLFile.cpp (L55)
  • .../WriteStatsGenOdfAngleFile.cpp (L56)

These fixes do the following:

  • L04, Align Sections (Feature Centroid): the highest-Z section is shifted when Use Reference Slice is on with a reference slice other than 0. A slice with no mask voxels gets a relative shift of 0, so its NaN centroid is never converted to an integer. With Use Reference Slice on, a Reference Slice at or above the Z dimension is rejected in preflight with -68073 (also BUG: C14: Parameter values not validated in preflight in 65 filters #1873 item AlignSectionsFeatureCentroidFilter#1). Align Sections (List) does not change.
  • L11, Compute Feature Clustering: the random reference uses all 1000 × 999 point pairs, so the RDF is no longer about 0.2 % high. This also closes BUG: Feature clustering RDF reference is not normalized to unit mass #1820.
  • L12, Compute Vector Colors: a trend that rounds to exactly 360° is red, not black.
  • L23, Quick Surface Mesh: transferred cell and feature data follow the Face Labels order in both the Direct and the Scanline algorithm. The existing exemplar archives hold the reversed 6.5.171 values. The exemplar tests swap the two halves of a face only where the archive's own transferred Feature Ids are exactly reversed, so any other mismatch still fails.
  • L28, Rotate Sample Reference Frame: for grid rotations (signed permutations, with or without a scale per axis), each output spacing comes from the source axis and scale that the classifier finds. This fixes three-axis cycles on anisotropic data. All other transformations keep the earlier spacing rule, so Apply Transformation To Geometry and Combine Transformation Matrices results do not change. Write OnScale Table File no longer fails with -12041 for these rotations. The V&V package records the change as deviation D2.
  • L29, VTK Rectilinear Grid Exporter: ASCII coordinates are written with full float precision, so a spacing of 1e-7 is no longer written as 0.
  • L54, Read EDAX OIMAnalysis Data (.h5) and Read Bruker Esprit Data (.h5): the selected scans are stacked in the chosen Stacking Order.
  • L55, Export INL File: every Laue class gets its TSL symmetry code in the data rows, not 999.
  • L56, Export StatsGenerator ODF Angle File: the Colon delimiter writes :.

2. Crashes and failures with a generic error

  • .../ReadStlFileFilter.cpp, .../utils/StlUtilities.cpp (L44)
  • .../AddBadDataFilter.cpp (L30)
  • .../CalculateTriangleGroupCurvatures.cpp (L39)
  • src/Plugins/ImageProcessing/src/ImageProcessing/Filters/Algorithms/ReadImage.cpp (L57)
  • .../Algorithms/ReadImageStack.cpp (L62)

These fixes do the following:

  • L44, Import STL File: a binary STL whose triangle count needs more bytes than the file has is rejected in preflight with -1108, before any allocation.
  • L30, Add Bad Data: a StringArray or NeighborList in the cell Attribute Matrix is rejected in preflight with -76239. It no longer throws.
  • L39, Find Feature Face Curvature: a Feature face of one triangle gets NaN outputs. The whole filter no longer fails with -2.
  • L57, Read Image: a multi-page TIFF page with a different number of bytes per pixel is rejected with -2007 before any of its bytes are copied. A page with a different layout but the same number of bytes per pixel is not detected; that is BUG: Fixes silent WRONG_DATA and CRASH bugs #1908 item L79.
  • L62, Read Image Stack: a slice with a different data type than the stack returns -23534 when the slice is read, before any typed access. The error names the slice and both types. Preflight still opens only file 0.

3. Input checks and messages

  • .../ReadDREAM3DFilter.cpp (L43)
  • .../ReadVtkStructuredPoints.cpp (L45)
  • .../WriteVtkRectilinearGridFilter.cpp, .../WriteVtkStructuredPointsFilter.cpp (L51, L52)
  • .../ExecuteProcess.cpp (L38)
  • src/simplnx/Utilities/ImageProcessing/ImageProcessingFilterUtilities.hpp (L58, L63)

These fixes do the following:

  • L43, Read DREAM.3D Data File: an Include List path that is not in the file produces warning -12.
  • L45, VTK STRUCTURED_POINTS Importer: 2D and 1D files load, because a singleton axis keeps one cell. That cell is centered on the VTK plane or line. A DIMENSIONS value of 0 is rejected with -10354, which names the axis.
  • L51, L52, VTK writers: arrays with 0 or more than 4 components are rejected in preflight (-12073, -12074). The legacy VTK SCALARS format allows only 1 to 4 components.
  • L38, Execute Process: separator spaces are no longer kept in the parsed arguments.
  • L58, L63, the seven projection filters with Perform In-Place on: warning -8343 names the objects that are discarded, by full path. This includes the other Cell Data arrays and the other children of the geometry, such as Cell Feature Data. One exception is documented: when the input array is outside Cell Data, a Cell Data array with the same name is not named (BUG: Fix MINOR defects found by the filter sweep and reviews #1917 item M31). The objects are still discarded. Keeping them will be an option in a separate pull request, after BUG: StringArray::deepCopy shares the string store with the original #1916.

Notes for reviewers

  • Behavior changes: several outputs now differ from DREAM.3D 6.5.171 on purpose: L04, L11, L12, L23, L28, L29, L54, L55 and L56. Each filter page says so.
  • New errors: pipelines that export arrays with more than 4 components through the legacy VTK writers now fail in preflight. This was chosen over writing FIELD data or only warning.
  • Not changed: Apply Transformation To Geometry with uneven scaling combined with an off-axis rotation keeps its rule-of-thumb output spacing. There is no correct result to test that against.
  • New error: Align Sections (Feature Centroid) now fails in preflight (-68073) when Use Reference Slice is on and the Reference Slice is at or above the Z dimension. Before, that input read past the end of an array.
  • Follow-up items: BUG: Fixes silent WRONG_DATA and CRASH bugs #1908 L77 (Relative Shifts content with a fixed reference slice), L79 (TIFF pages with the same number of bytes per pixel), and BUG: Fix MINOR defects found by the filter sweep and reviews #1917 M31 (the projection warning excludes the input by name).

Test plan

  • Each new regression test fails on develop and passes on this branch. The failing logs were recorded for every item.
  • Full serial ctest on macOS arm64 (ci-macos-arm64): 3537/3537 passed at ce1472576.
  • The commits from the Linux lane were built and tested serially with clang-16 on Ubuntu 22.04 (ci-linux-x64).
  • Prebuilt Pipelines: Parameter Keys Are Current passes.
  • CI passes on all supported platforms.

Related Issue

Related to #1908
Related to #1873
Closes #1820

@imikejackson imikejackson self-assigned this Nov 4, 2026

@jmarquisbq jmarquisbq left a comment •

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed all 98 changed files at 45fb158f538b64cea6766f9520ebb61918fffa76.

Findings

  • Critical · Guard empty mask slices before converting centroid shifts. The new row-0 path computes a shift for the highest-Z slice when a lower slice is the reference. If the top slice has no valid mask voxels, its centroid is NaN. Conversion to an integer shift has undefined behavior. Define the behavior for an empty source or reference slice before computing shifts. Test Direct and Scanline with diagnostics enabled and disabled. AlignSectionsFeatureCentroid.cpp:130

  • Important · Make projection copies independent and propagate child copy failures. The new option exposes two existing copy defects through seven filters. Editing a copied StringArray changes the input. A numeric child copy can also fail while the copy action returns success with an incomplete matrix. A local probe reproduced both outcomes through the action used by the projection. Fix recursive error propagation and string ownership at the copy boundary, or reject contents that the option cannot copy safely. ImageProcessingFilterUtilities.hpp:800, AttributeMatrix.cpp:61, StringArray.cpp:99

  • Important · Validate TIFF scalar type and component count before writing each page. The byte-count check prevents the reported overread. Different layouts with the same byte width still pass. A local probe imported an RGBA8 first page followed by a grayscale Float32 page. Float32 1.0 became RGBA bytes (0, 0, 128, 63). Compare the selected page's dimensions, component count, and scalar type with page 0 before invoking the row callback. ReadImage.cpp:362

  • Important · Carry the classified axis mapping into spacing calculation. The helper classifies normalized columns and then selects spacing from the largest entry in each raw row. The two choices can disagree under uneven scaling. With x′ = x + 5y, y′ = 100000y, and input spacing (2,3,5), the probe returned X spacing 300000 and one X cell. The classifier assigned input X to output X. Carry the source-axis index and scale out of classification and use them directly. Exercise this case through Apply Transformation to Geometry as well as the helper. ImageRotationUtilities.cpp:94, GeometryTransformation.cpp:229

  • Important · Preserve the physical position of VTK planes and lines. Singleton-axis handling preserves tuple counts but shifts voxel centers by half the spacing on each collapsed axis. VTK placed the test plane at Z=10. With Z spacing 4, simplnx imported it with a voxel center at Z=12. Define how the importer places a lower-dimensional grid. Centering the one-voxel slab on the source plane would preserve its location. Test physical coordinates alongside dimensions and values. ReadVtkStructuredPoints.cpp:839

  • Important · Define which centroid diagnostics can be replayed as relative shifts. With a fixed reference, Feature Centroid stores absolute offsets in Relative Shifts. List treats those values as adjacent-slice deltas. The new fixture's offsets [1, 0, -1, 0] replay as [1, 1, 0, 0] in Relative mode. The PR makes the first diagnostic row name the fixed reference, while later rows still name the previous slice. Encode adjacent deltas consistently, or document Cumulative as the replay format for fixed-reference output. Add a producer-to-consumer test. AlignSectionsFeatureCentroid.cpp:182, AlignSectionsList.cpp:51

  • Minor · Scope the V&V equivalence claim to the four tested fixtures. The deviation page describes principal-axis rotations as the supported domain. The filter also accepts proper signed permutations such as the three-cycle. Suggested wording follows. “The four isotropic principal-axis fixtures produced identical dimensions and voxel values in simplnx and 6.5.171. D2 verifies the anisotropic three-cycle against an analytical result.” RotateSampleRefFrameFilter.md:7

Architectural implications

  • Projection changes data ownership. Pipeline preflight applies CopyDataObjectAction, which deep-copies populated stores in both action modes. A preflight can therefore copy full feature matrices to model the output. In-place execution moves objects before a geometry swap that uses several actions. Use metadata-only copies in preflight and define what remains visible after a copy or swap fails. The projection option warrants its own PR because the copy contract affects every projection and other callers of the action. CopyDataObjectAction.cpp:29, PipelineFilter.cpp:196

  • Image Stack depends on another filter's action order. The validation scan runs ReadImageFilter preflight for each retained slice and assumes its first CreateArrayAction describes the image. The execution path opens and checks the files again. This adds header I/O and leaves a gap if a file changes between validation and decoding. Share a typed layout descriptor between validation and decoding. Check the actual slice before the typed copy. ReadImageStackFilter.cpp:475

  • Ordering is part of the data contract. Quick Surface Mesh coordinates triangle winding, face labels, and transferred tuples through separate decisions in Direct and Scanline. The two-cell test provides an independent oracle. A single ordered pair of sides per face would help keep those decisions consistent. In Align Sections (List), applying row 0 also changes saved pipelines with a nonzero first row. Treat this as a compatibility decision and test replay of a saved pipeline. QuickSurfaceMeshDirect.cpp:1175, AlignSectionsListFilter.md:21

Verification

  • Windows Release passed 241 selected CTest cases, including all 30 added or renamed cases and the prebuilt pipeline key check. An earlier run overlapped plugin linking. Both checks affected by stale binaries passed after rebuilding.
  • Clang-format 19.1.1 passed all 65 changed C++ files. A scoped clang-tidy 19.1.5 run reported diagnostics for direct includes, naming, constness, and unchecked optional access.
  • Debug, disk-backed out-of-core, and full-suite validation remain open.

REQUEST CHANGES. Fix the undefined centroid conversion and the paths that silently alter, omit, misinterpret, or misplace output data.

@imikejackson imikejackson changed the title BUG: Fix 21 filter defects that DREAM3D-NX shares with DREAM.3D 6.5.171 BUG: Fix 21 filter defects ported from 'SIMPL' and 'DREAM3D' Nov 4, 2026
imikejackson added a commit to imikejackson/simplnx that referenced this pull request Nov 4, 2026
Review of BlueQuartzSoftware#1915: the Copy Non-Cell Attribute Matrices option went
beyond the filed defect. It exposed the StringArray deep-copy defect
(BlueQuartzSoftware#1916) and copy failures that are not reported, and it made preflight
copy full feature matrices. The option moves to a separate pull
request.

Remove the parameter from the seven projection filters, the move and
copy actions in the shared helper, the parameter version bump, and the
pipeline and Python example changes. The projection filter sources and
the shipped pipeline are identical to develop again.

Keep the fix of the filed defect: with Perform In-Place on, warning
-8343 names every discarded Cell Data array (including one with the
output name) and every other child of the geometry, such as Cell
Feature Data, by full path. The consumed input is not listed, also
when it is not in Cell Data. Tests for Maximum and Standard Deviation
check the warning text.

Fixes BlueQuartzSoftware#1908 item L58
Fixes BlueQuartzSoftware#1908 item L63

Signed-off-by: Michael Jackson <mike.jackson@bluequartz.net>
imikejackson added a commit to imikejackson/simplnx that referenced this pull request Nov 4, 2026
Review of BlueQuartzSoftware#1915: commit 035079b also made Align Sections (List)
apply row 0, so a Feature Centroid result could be replayed. That went
beyond the L04 defect, changed the output of saved pipelines with a
nonzero first row, and exposed a separate defect: with a fixed
reference slice, Feature Centroid writes reference offsets into the
Relative Shifts array (BlueQuartzSoftware#1908 item L77).

Withdraw that part. The Align Sections (List) source, docs and tests
are identical to develop again, and Align_Sections_User_Input gives
its develop output. List leaves row 0 at 0, so the shared transfer
still skips it. The Feature Centroid fix for the top slice stays.

Fixes BlueQuartzSoftware#1908 item L04

Signed-off-by: Michael Jackson <mike.jackson@bluequartz.net>
imikejackson added a commit to imikejackson/simplnx that referenced this pull request Nov 4, 2026
Review of BlueQuartzSoftware#1915: a slice with no mask voxels has a NaN centroid, and
converting a NaN shift to int64 is undefined behavior. Commit
035079b made the top slice reach this conversion when Use Reference
Slice is on with a reference slice other than 0; other slices already
did on develop.

A shift is now computed only when both centroids it uses are finite.
Otherwise the relative shift is 0, so in consecutive mode the slice
keeps the cumulative shift of the previous slice. This applies to all
in-core and out-of-core loops, with and without the diagnostic arrays.
The NaN info messages now include the slice number.

Add tests with an empty top slice (reference slice 1) and an empty
middle slice (consecutive mode), with hand-calculated shifts, for the
in-core and out-of-core paths with Store Alignment Shifts on and off.
UBSan reports the undefined conversion before the fix.

Fixes BlueQuartzSoftware#1908 item L04

Signed-off-by: Michael Jackson <mike.jackson@bluequartz.net>
imikejackson added a commit to imikejackson/simplnx that referenced this pull request Nov 4, 2026
Review of BlueQuartzSoftware#1915: the earlier fix ran Read Image preflight for every
slice at the start of execute and trusted the type of its first
CreateArrayAction. That added header reads and did not protect the
typed access if a file changed between the check and the read.

Remove the pre-scan. The algorithm now compares the decoded slice's
data type with the stack type right after Read Image returns, before
resampling, grayscale conversion, flips, the typed access and the
copy, and returns -23534 with the slice number, file name and both
types. Preflight still opens only file 0. Slices before the bad one
may already be written; the documentation says so, and the test no
longer requires an untouched output after the error.

Fixes BlueQuartzSoftware#1908 item L62

Signed-off-by: Michael Jackson <mike.jackson@bluequartz.net>
imikejackson added a commit to imikejackson/simplnx that referenced this pull request Nov 4, 2026
…-rotation spacing

Review of BlueQuartzSoftware#1915: CreateRotationArgs classified a grid rotation from
the normalized columns, but then chose each output spacing from the
largest entry of the raw matrix row. Under uneven scaling with a small
shear the two disagree. For x' = x + 5y, y' = 100000y and spacing
(2,3,5), the classifier maps X to X, but the output got X spacing
300000 and one X cell.

The classifier now records, for each column, its output row and its
scale, and the output spacing comes directly from that mapping. The
non-grid path does not change.

Add closed-form tests of this matrix through the helper and through
Apply Transformation to Geometry (Direct and Scanline): spacing
(2, 300000, 5) and dimensions (17, 2, 2). Use the reviewer's wording
for the V&V deviation: only the four isotropic principal-axis fixtures
are claimed identical to 6.5.171.

Fixes BlueQuartzSoftware#1908 item L28

Signed-off-by: Michael Jackson <mike.jackson@bluequartz.net>
imikejackson added a commit to imikejackson/simplnx that referenced this pull request Nov 4, 2026
…tion

Review of BlueQuartzSoftware#1915: a DIMENSIONS value of 1 gives that axis one cell, but
the origin stayed on the VTK point, so the voxel center moved by half
the spacing. A plane at Z = 10 with Z spacing 4 was imported with its
voxel center at Z = 12.

On every axis with one point, the origin of the cell and point Image
Geometries is now the VTK origin minus half the spacing, so the voxel
is centered on the plane or line. Axes with more than one point do not
change.

Add tests for a plane and for X, Y and Z lines that check dimensions,
values, origin and voxel centers of both geometries, in preflight and
execute.

Fixes BlueQuartzSoftware#1908 item L45

Signed-off-by: Michael Jackson <mike.jackson@bluequartz.net>
imikejackson added a commit to imikejackson/simplnx that referenced this pull request Nov 4, 2026
Review of BlueQuartzSoftware#1915: the -2007 messages and the loop comment written for
L57 said that every page must share dimensions, component count and
scalar type, which read as if all three were checked. The code
compares only the number of bytes per pixel and the dimensions of each
page with page 0, because IImageIO gives metadata only for page 0.

Each -2007 message now says what was found (a different number of
bytes per pixel, or different dimensions) and what is required. The
comment and the filter documentation say that a page with a different
layout but the same number of bytes per pixel is not detected. That
case is tracked as BlueQuartzSoftware#1908 item L79.

Fixes BlueQuartzSoftware#1908 item L57

Signed-off-by: Michael Jackson <mike.jackson@bluequartz.net>
@imikejackson

Copy link
Copy Markdown
Contributor Author

Thanks for the thorough review. Seven REV: commits address it. We also removed the parts of the PR that went beyond the defects filed in #1908.

Findings

  • Critical · Empty mask slices (Feature Centroid): fixed in 276e696. A shift is computed only when both centroids are finite. Otherwise the relative shift is 0. This applies to every in-core and out-of-core loop, with and without the diagnostic arrays. The tests cover an empty top slice with reference slice 1 and an empty middle slice in consecutive mode, on both paths. UBSan flags the conversion before the fix.
  • Important · Projection copies: the Copy Non-Cell Attribute Matrices option is removed from this PR (908ea1d) and will come in a separate PR after BUG: StringArray::deepCopy shares the string store with the original #1916. This PR now keeps only the fix of the filed defect. Warning -8343 names every discarded object by full path. The projection filter sources and the shipped pipeline are identical to develop again.
  • Important · TIFF scalar type and component count: out of scope for L57, which was the overread. A real fix needs per-page metadata from IImageIO. It is tracked as BUG: Fixes silent WRONG_DATA and CRASH bugs #1908 item L79. 2b3c895 corrects the -2007 messages, the comment and the docs, so they no longer claim that the component count and scalar type are checked.
  • Important · Classified axis mapping: fixed in 44b3f3d. The spacing comes directly from the source axis and scale found by the classifier. Your matrix is tested through the helper and through Apply Transformation to Geometry (Direct and Scanline): spacing (2, 300000, 5), dimensions (17, 2, 2).
  • Important · VTK planes and lines: fixed in de633be. On an axis with one point, the origin is moved by half the spacing, so the voxel is centered on the plane. Your case now gives a voxel center at Z = 10. The tests check the origin and voxel centers of the cell and point geometries.
  • Important · Replaying centroid diagnostics: f819b13 withdraws the Align Sections (List) row-0 change. List is identical to develop again, so saved pipelines do not change. The Relative Shifts content in fixed-reference mode is tracked as BUG: Fixes silent WRONG_DATA and CRASH bugs #1908 item L77.
  • Minor · V&V equivalence claim: your wording is used word for word (44b3f3d).

Architectural notes

  • Projection data ownership: this no longer applies to this PR, because the option is removed. The follow-up PR will use metadata-only copies in preflight.
  • Image Stack action order: the pre-scan is removed (86f72ab). The algorithm checks the decoded slice's data type where it reads the slice, before any typed access, and returns -23534.
  • Ordering contract: we agree about Quick Surface Mesh. The List compatibility concern is resolved by the revert.

Verification

  • Every commit was reviewed in alternating rounds by two independent reviewers until two consecutive rounds were clean.
  • Full serial ctest on macOS arm64: 3543/3543 passed.
  • The R4 to R7 commits were also built and tested with clang-16 on Ubuntu 22.04.

@imikejackson
imikejackson requested review from jmarquisbq and a balanced review from Copilot and removed request for Copilot November 4, 2026 23:34
@jmarquisbq

Copy link
Copy Markdown
Contributor

Rechecked the two findings from the review at 2b3c8951f1683b313d44da10316e97a2c7624a1c against the response, develop, the issue tracker, and the open pull requests.

  • Request changes · One-slice reference index. The missing upper-bound validation is already tracked in BUG: C14: Parameter values not validated in preflight in 65 filters #1873. On develop, the shift loop starts at 1 and skips a one-slice volume. This PR starts at 0 when Use Reference Slice is enabled with a nonzero reference. A one-slice geometry with reference index 1 passes preflight and now reads beyond the centroid vector. A Windows AddressSanitizer probe confirmed the heap-buffer-overflow. Please reject an out-of-range reference or otherwise prevent this new access. Changed loop

  • Correction · Projection warning. The same-name comparison exists on develop and is already tracked as BUG: Fix MINOR defects found by the filter sweep and reviews #1917 item M31. I withdraw it as an Important blocking finding for this PR. The new documentation says warning -8343 names every discarded object. Please qualify that sentence while M31 remains open. Projection documentation

I checked the changed-file lists of the other open simplnx PRs. I fetched and compared the heads locally where GitHub could not show the full diff. #1888 changes the shared image-processing utility for other algorithms and retains the projection comparison. #1904 renames the plugin and retains that comparison. I found no open simplnx PR that resolves either condition. I also checked open DREAM3DNX and SimplnxOoc PR descriptions for these issues.

The earlier response addresses the previous review and explains the narrower scope. The revised verdict remains REQUEST CHANGES for the newly reachable one-slice out-of-bounds read.

@imikejackson
imikejackson enabled auto-merge (squash) November 5, 2026 15:37
@imikejackson
imikejackson force-pushed the topic/errata_1908_fixes branch from 2b3c895 to 7ef155a Compare November 5, 2026 15:38
imikejackson added a commit to imikejackson/simplnx that referenced this pull request Nov 5, 2026
Review of BlueQuartzSoftware#1915: the Copy Non-Cell Attribute Matrices option went
beyond the filed defect. It exposed the StringArray deep-copy defect
(BlueQuartzSoftware#1916) and copy failures that are not reported, and it made preflight
copy full feature matrices. The option moves to a separate pull
request.

Remove the parameter from the seven projection filters, the move and
copy actions in the shared helper, the parameter version bump, and the
pipeline and Python example changes. The projection filter sources and
the shipped pipeline are identical to develop again.

Keep the fix of the filed defect: with Perform In-Place on, warning
-8343 names every discarded Cell Data array (including one with the
output name) and every other child of the geometry, such as Cell
Feature Data, by full path. The consumed input is not listed, also
when it is not in Cell Data. Tests for Maximum and Standard Deviation
check the warning text.

Fixes BlueQuartzSoftware#1908 item L58
Fixes BlueQuartzSoftware#1908 item L63

Signed-off-by: Michael Jackson <mike.jackson@bluequartz.net>
imikejackson added a commit to imikejackson/simplnx that referenced this pull request Nov 5, 2026
Review of BlueQuartzSoftware#1915: commit 035079b also made Align Sections (List)
apply row 0, so a Feature Centroid result could be replayed. That went
beyond the L04 defect, changed the output of saved pipelines with a
nonzero first row, and exposed a separate defect: with a fixed
reference slice, Feature Centroid writes reference offsets into the
Relative Shifts array (BlueQuartzSoftware#1908 item L77).

Withdraw that part. The Align Sections (List) source, docs and tests
are identical to develop again, and Align_Sections_User_Input gives
its develop output. List leaves row 0 at 0, so the shared transfer
still skips it. The Feature Centroid fix for the top slice stays.

Fixes BlueQuartzSoftware#1908 item L04

Signed-off-by: Michael Jackson <mike.jackson@bluequartz.net>
imikejackson added a commit to imikejackson/simplnx that referenced this pull request Nov 5, 2026
Review of BlueQuartzSoftware#1915: a slice with no mask voxels has a NaN centroid, and
converting a NaN shift to int64 is undefined behavior. Commit
035079b made the top slice reach this conversion when Use Reference
Slice is on with a reference slice other than 0; other slices already
did on develop.

A shift is now computed only when both centroids it uses are finite.
Otherwise the relative shift is 0, so in consecutive mode the slice
keeps the cumulative shift of the previous slice. This applies to all
in-core and out-of-core loops, with and without the diagnostic arrays.
The NaN info messages now include the slice number.

Add tests with an empty top slice (reference slice 1) and an empty
middle slice (consecutive mode), with hand-calculated shifts, for the
in-core and out-of-core paths with Store Alignment Shifts on and off.
UBSan reports the undefined conversion before the fix.

Fixes BlueQuartzSoftware#1908 item L04

Signed-off-by: Michael Jackson <mike.jackson@bluequartz.net>
imikejackson added a commit to imikejackson/simplnx that referenced this pull request Nov 5, 2026
Review of BlueQuartzSoftware#1915: the earlier fix ran Read Image preflight for every
slice at the start of execute and trusted the type of its first
CreateArrayAction. That added header reads and did not protect the
typed access if a file changed between the check and the read.

Remove the pre-scan. The algorithm now compares the decoded slice's
data type with the stack type right after Read Image returns, before
resampling, grayscale conversion, flips, the typed access and the
copy, and returns -23534 with the slice number, file name and both
types. Preflight still opens only file 0. Slices before the bad one
may already be written; the documentation says so, and the test no
longer requires an untouched output after the error.

Fixes BlueQuartzSoftware#1908 item L62

Signed-off-by: Michael Jackson <mike.jackson@bluequartz.net>
imikejackson added a commit to imikejackson/simplnx that referenced this pull request Nov 5, 2026
…-rotation spacing

Review of BlueQuartzSoftware#1915: CreateRotationArgs classified a grid rotation from
the normalized columns, but then chose each output spacing from the
largest entry of the raw matrix row. Under uneven scaling with a small
shear the two disagree. For x' = x + 5y, y' = 100000y and spacing
(2,3,5), the classifier maps X to X, but the output got X spacing
300000 and one X cell.

The classifier now records, for each column, its output row and its
scale, and the output spacing comes directly from that mapping. The
non-grid path does not change.

Add closed-form tests of this matrix through the helper and through
Apply Transformation to Geometry (Direct and Scanline): spacing
(2, 300000, 5) and dimensions (17, 2, 2). Use the reviewer's wording
for the V&V deviation: only the four isotropic principal-axis fixtures
are claimed identical to 6.5.171.

Fixes BlueQuartzSoftware#1908 item L28

Signed-off-by: Michael Jackson <mike.jackson@bluequartz.net>
imikejackson added a commit to imikejackson/simplnx that referenced this pull request Nov 5, 2026
…tion

Review of BlueQuartzSoftware#1915: a DIMENSIONS value of 1 gives that axis one cell, but
the origin stayed on the VTK point, so the voxel center moved by half
the spacing. A plane at Z = 10 with Z spacing 4 was imported with its
voxel center at Z = 12.

On every axis with one point, the origin of the cell and point Image
Geometries is now the VTK origin minus half the spacing, so the voxel
is centered on the plane or line. Axes with more than one point do not
change.

Add tests for a plane and for X, Y and Z lines that check dimensions,
values, origin and voxel centers of both geometries, in preflight and
execute.

Fixes BlueQuartzSoftware#1908 item L45

Signed-off-by: Michael Jackson <mike.jackson@bluequartz.net>
imikejackson added a commit to imikejackson/simplnx that referenced this pull request Nov 5, 2026
Review of BlueQuartzSoftware#1915: the -2007 messages and the loop comment written for
L57 said that every page must share dimensions, component count and
scalar type, which read as if all three were checked. The code
compares only the number of bytes per pixel and the dimensions of each
page with page 0, because IImageIO gives metadata only for page 0.

Each -2007 message now says what was found (a different number of
bytes per pixel, or different dimensions) and what is required. The
comment and the filter documentation say that a page with a different
layout but the same number of bytes per pixel is not detected. That
case is tracked as BlueQuartzSoftware#1908 item L79.

Fixes BlueQuartzSoftware#1908 item L57

Signed-off-by: Michael Jackson <mike.jackson@bluequartz.net>
@imikejackson
imikejackson disabled auto-merge November 5, 2026 16:39
imikejackson and others added 16 commits November 13, 2026 11:39
…er is chosen

Delimiter choice 3 is labeled Colon in the parameter and in the file
header, but the rows were written with a double quote (") between the
values. DREAM.3D 6.5.171 has the same defect (errata KI-015).

The delimiter table now uses : for that choice. No reader takes the
delimiter from the header line, and the StatsGenerator header line
"Angle Count:N" is read before the data rows, so the change breaks no
reader.

Add a test with two masked tuples in two phases that checks the exact
colon-separated rows (hand-converted to degrees) in the Direct and
Scanline algorithms.

Fixes BlueQuartzSoftware#1908 item L56

Signed-off-by: Mike Jackson <mike.jackson@bluequartz.net>
…ore copying it

In a multi-page TIFF, the reader detected that a later page had a
different number of components than page 0 (for example an RGB page 0
and a grayscale page 1), but it still copied the page with the pixel
size of page 0. The copy read past the end of the narrower page buffer
and wrote wrong values into that Z slice, and the -2007 error came only
after the copy. DREAM.3D 6.5.171 (errata KI-053) had the same problem
in its ITK reader.

The reader now returns -2007 as soon as it reads such a page, before
any crop, conversion or write to the data store. The late -2007 check
now reports only dimension mismatches.

Add a test that writes a real two-page TIFF (RGB 255, then grayscale
127) with libtiff and checks, with and without conversion to uint16,
that page 0 is read, error -2007 is returned and slice 1 is untouched.

Fixes BlueQuartzSoftware#1908 item L57

Signed-off-by: Mike Jackson <mike.jackson@bluequartz.net>
…d name dropped objects

With Perform In-Place on, the shared projection helper replaces the
whole input geometry. Every other Attribute Matrix on it (for example
Cell Feature Data and Cell Ensemble Data) was deleted, and warning -8343
named only the dropped Cell Data arrays. DREAM.3D 6.5.171 deleted the
other arrays too (errata KI-009).

All seven projection filters (Maximum, Minimum, Mean, Median, Sum,
Standard Deviation and Binary) get a new parameter, Copy Non-Cell
Attribute Matrices (copy_non_cell_attribute_matrices), default false,
parameter version 2:

- false: unchanged behavior. In-place warning -8343 now lists every
  discarded Cell Data array and every other discarded geometry child.
- true, in-place: the non-cell Attribute Matrices are moved to the
  projected geometry. No arrays are copied.
- true, new geometry: the non-cell Attribute Matrices are deep-copied
  into the created geometry. String arrays in them currently share
  their values with the input (StringArray::deepCopy shares its store;
  separate issue).

Older pipelines without the key load with false.

Add tests for Maximum and Standard Deviation that cover both modes, the
omitted, false and true option values, and an extra Cell Data array,
with hand-written fixture values, and tests that version-1 JSON loads
with false.

Fixes BlueQuartzSoftware#1908 item L58
Fixes BlueQuartzSoftware#1908 item L63

Signed-off-by: Mike Jackson <mike.jackson@bluequartz.net>
…ad of throwing

The stack array takes its data type from file 0. A later slice with a
different type (for example uint16 in a uint8 stack) made the typed
array lookup throw std::bad_cast after earlier slices were already
copied, and the user got the generic error -2. DREAM.3D 6.5.171 could
crash or produce zero slices in its ITK reader (errata KI-051).

At the start of execute, before any slice is decoded or written, the
filter now reads the header of every kept slice and returns -23534,
naming the slice, its type and the stack type, when a type differs.
Preflight still opens only file 0, so editing a pipeline with a large
stack stays fast. With an explicit output data type, a mixed stack is
converted and imported as before.

Add a test with generated uint8 and uint16 TIFF slices in both orders:
without conversion, preflight passes and execute returns -23534 with
the stack left at zero; with uint16 conversion, both slices import
(10 -> 2570, 1000); a Z crop that keeps only slice 0 succeeds.

Fixes BlueQuartzSoftware#1908 item L62

Signed-off-by: Mike Jackson <mike.jackson@bluequartz.net>
Review of BlueQuartzSoftware#1915: the Copy Non-Cell Attribute Matrices option went
beyond the filed defect. It exposed the StringArray deep-copy defect
(BlueQuartzSoftware#1916) and copy failures that are not reported, and it made preflight
copy full feature matrices. The option moves to a separate pull
request.

Remove the parameter from the seven projection filters, the move and
copy actions in the shared helper, the parameter version bump, and the
pipeline and Python example changes. The projection filter sources and
the shipped pipeline are identical to develop again.

Keep the fix of the filed defect: with Perform In-Place on, warning
-8343 names every discarded Cell Data array (including one with the
output name) and every other child of the geometry, such as Cell
Feature Data, by full path. The consumed input is not listed, also
when it is not in Cell Data. Tests for Maximum and Standard Deviation
check the warning text.

Fixes BlueQuartzSoftware#1908 item L58
Fixes BlueQuartzSoftware#1908 item L63

Signed-off-by: Michael Jackson <mike.jackson@bluequartz.net>
Review of BlueQuartzSoftware#1915: commit 035079b also made Align Sections (List)
apply row 0, so a Feature Centroid result could be replayed. That went
beyond the L04 defect, changed the output of saved pipelines with a
nonzero first row, and exposed a separate defect: with a fixed
reference slice, Feature Centroid writes reference offsets into the
Relative Shifts array (BlueQuartzSoftware#1908 item L77).

Withdraw that part. The Align Sections (List) source, docs and tests
are identical to develop again, and Align_Sections_User_Input gives
its develop output. List leaves row 0 at 0, so the shared transfer
still skips it. The Feature Centroid fix for the top slice stays.

Fixes BlueQuartzSoftware#1908 item L04

Signed-off-by: Michael Jackson <mike.jackson@bluequartz.net>
Review of BlueQuartzSoftware#1915: a slice with no mask voxels has a NaN centroid, and
converting a NaN shift to int64 is undefined behavior. Commit
035079b made the top slice reach this conversion when Use Reference
Slice is on with a reference slice other than 0; other slices already
did on develop.

A shift is now computed only when both centroids it uses are finite.
Otherwise the relative shift is 0, so in consecutive mode the slice
keeps the cumulative shift of the previous slice. This applies to all
in-core and out-of-core loops, with and without the diagnostic arrays.
The NaN info messages now include the slice number.

Add tests with an empty top slice (reference slice 1) and an empty
middle slice (consecutive mode), with hand-calculated shifts, for the
in-core and out-of-core paths with Store Alignment Shifts on and off.
UBSan reports the undefined conversion before the fix.

Fixes BlueQuartzSoftware#1908 item L04

Signed-off-by: Michael Jackson <mike.jackson@bluequartz.net>
Review of BlueQuartzSoftware#1915: the earlier fix ran Read Image preflight for every
slice at the start of execute and trusted the type of its first
CreateArrayAction. That added header reads and did not protect the
typed access if a file changed between the check and the read.

Remove the pre-scan. The algorithm now compares the decoded slice's
data type with the stack type right after Read Image returns, before
resampling, grayscale conversion, flips, the typed access and the
copy, and returns -23534 with the slice number, file name and both
types. Preflight still opens only file 0. Slices before the bad one
may already be written; the documentation says so, and the test no
longer requires an untouched output after the error.

Fixes BlueQuartzSoftware#1908 item L62

Signed-off-by: Michael Jackson <mike.jackson@bluequartz.net>
…-rotation spacing

Review of BlueQuartzSoftware#1915: CreateRotationArgs classified a grid rotation from
the normalized columns, but then chose each output spacing from the
largest entry of the raw matrix row. Under uneven scaling with a small
shear the two disagree. For x' = x + 5y, y' = 100000y and spacing
(2,3,5), the classifier maps X to X, but the output got X spacing
300000 and one X cell.

The classifier now records, for each column, its output row and its
scale, and the output spacing comes directly from that mapping. The
non-grid path does not change.

Add closed-form tests of this matrix through the helper and through
Apply Transformation to Geometry (Direct and Scanline): spacing
(2, 300000, 5) and dimensions (17, 2, 2). Use the reviewer's wording
for the V&V deviation: only the four isotropic principal-axis fixtures
are claimed identical to 6.5.171.

Fixes BlueQuartzSoftware#1908 item L28

Signed-off-by: Michael Jackson <mike.jackson@bluequartz.net>
…tion

Review of BlueQuartzSoftware#1915: a DIMENSIONS value of 1 gives that axis one cell, but
the origin stayed on the VTK point, so the voxel center moved by half
the spacing. A plane at Z = 10 with Z spacing 4 was imported with its
voxel center at Z = 12.

On every axis with one point, the origin of the cell and point Image
Geometries is now the VTK origin minus half the spacing, so the voxel
is centered on the plane or line. Axes with more than one point do not
change.

Add tests for a plane and for X, Y and Z lines that check dimensions,
values, origin and voxel centers of both geometries, in preflight and
execute.

Fixes BlueQuartzSoftware#1908 item L45

Signed-off-by: Michael Jackson <mike.jackson@bluequartz.net>
Review of BlueQuartzSoftware#1915: the -2007 messages and the loop comment written for
L57 said that every page must share dimensions, component count and
scalar type, which read as if all three were checked. The code
compares only the number of bytes per pixel and the dimensions of each
page with page 0, because IImageIO gives metadata only for page 0.

Each -2007 message now says what was found (a different number of
bytes per pixel, or different dimensions) and what is required. The
comment and the filter documentation say that a page with a different
layout but the same number of bytes per pixel is not detected. That
case is tracked as BlueQuartzSoftware#1908 item L79.

Fixes BlueQuartzSoftware#1908 item L57

Signed-off-by: Michael Jackson <mike.jackson@bluequartz.net>
…Slice in preflight

Preflight rejected only a negative Reference Slice. Since the L04 fix, the
shift loops start at slice 0 when Use Reference Slice is on with a nonzero
reference, so a one-slice geometry with Reference Slice 1 read past the end
of the centroid vectors. A value at or above the Z dimension already did the
same on develop for two or more slices.

With Use Reference Slice on, preflight now rejects a Reference Slice that is
at or above the Z dimension with error -68073. The geometry is read even when
Store Alignment Shifts is off. The test covers one and three slices, the
values Z and Z-1, and both settings of Use Reference Slice and Store
Alignment Shifts.

Fixes BlueQuartzSoftware#1908 item L04
Fixes BlueQuartzSoftware#1873 item AlignSectionsFeatureCentroidFilter#1

Signed-off-by: Michael Jackson <mike.jackson@bluequartz.net>
The seven projection pages said that warning -8343 names every discarded
object. When the input array is outside Cell Data, a Cell Data array with
the same name as the input is discarded but is not named, because the Cell
Data loop skips children by name. The pages now state this exception.
The code is unchanged; BlueQuartzSoftware#1917 item M31 tracks the fix.

Signed-off-by: Michael Jackson <mike.jackson@bluequartz.net>
…mments

Address the review of ce14725:

- 1.1: Remove UnitTest::LoadPlugins() from 23 new test cases that
  construct their filter or utility directly. None of them looks up a
  filter by UUID, loads a pipeline, runs a filter that gets subfilters
  from the filter list or uses PreferencesSentinel. The three new
  projection-filter cases that use PreferencesSentinel keep the call:
  the sentinel changes the preferences of the Application instance that
  LoadPlugins() creates, and the cases crash without it. The coding
  knowledge base rule now names this case.
- 2.1: Tag the five new ImageRotationUtilities cases [simplnx]. The new
  AlignSectionsFeatureCentroid and FeatureFaceCurvature cases now start
  with [SimplnxCore].
- 2.2: Document the Reference Slice bound and error -68073.
- 2.3: Split four comment sentences that were longer than 25 words.

Signed-off-by: Michael Jackson <mike.jackson@bluequartz.net>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
develop renamed the ImageProcessing plugin to NXImageProcessing (BlueQuartzSoftware#1904).
Four test cases that this branch added still used the old ImageProcessing
prefix and tag. Rename them to match the rest of the plugin's tests.

Signed-off-by: Michael Jackson <mike.jackson@bluequartz.net>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Michael Jackson <mike.jackson@bluequartz.net>
…r-bin test

develop added the four-bin control test (BlueQuartzSoftware#1949) with reference-pair
counts that omit pairs involving sample point 0, matching the old
normalization. This branch counts those pairs, so the RDF values in the
test were about 0.2% too high.

The new counts come from an independent lattice-cell population
enumeration of the seed-5489 sample. It sums 999000 ordered pairs.

Signed-off-by: Michael Jackson <mike.jackson@bluequartz.net>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Michael Jackson <mike.jackson@bluequartz.net>

@jmarquisbq jmarquisbq left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

PR 1915 follow-up review

Reviewed fe785bcdd8172172d698942536ef497cc152c5cb. I checked the three files changed since 63571cb20 and carried forward the prior review of the unchanged diffs. Together these checks cover all 83 PR files.

1. Requested fixes

  • 1.1 The core rotation test and the three projection warning tests no longer load unrelated plugins. The projection fixtures initialize Application before constructing PreferencesSentinel. The coding knowledge base now describes that requirement.
  • 1.2 The eight regression cases have the requested core or plugin tags.
  • 1.3 The reference-slice documentation gives the allowed range and error -68073.
  • 1.4 All four flagged comment sentences were shortened.
  • 1.5 The same-byte-width TIFF layout limitation remains tracked as #1908 item L79. It exists on the base revision. This PR fixes the overread when byte widths differ.

2. Architecture and validation

  • 2.1 The final follow-up removes an unnecessary dependency on plugin registration in the tests. The affected calls, singleton lifetime, preferences restoration, and documented fixture contract agree. The follow-up changes test setup and documentation. No actionable implementation or architectural findings remain in scope.
  • 2.2 The exact final patch passed its Release build and all three selected projection regressions on CI-Ubuntu-20. The prior head passed 191 selected Release tests on Helium. The final commit matches the patch tested on CI-Ubuntu-20.
  • 2.3 Clang-format 19.1.1 passed all 51 C++ files in the PR at this head.
  • 2.4 GitHub reports ACTION_REQUIRED for the current check. This review did not run Debug, a full suite, or disk-backed out-of-core validation.

3. Verdict

APPROVE. The requested fixes within this PR's scope are addressed.

Create the Application before constructing PreferencesSentinel in the
three projection warning tests. Update the coding knowledge base to
explain when tests need application initialization or plugin loading.

Signed-off-by: Jessica Marquis <jessica.marquis@bluequartz.net>
Signed-off-by: Jessica Marquis <jessica.marquis@bluequartz.net>

@jmarquisbq jmarquisbq left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

PR 1915 follow-up review

Reviewed the rebase at 4830a1716 and the include cleanup at 92823db33bb0c32be9dc73bb43e5790322bb34dd. The rebase preserves the requested fixes, including the projection test setup and the reference-slice validation and documentation.

1. Findings

  • 1.1 Resolved. Removed the unused <stdexcept> include from WriteINLFileTest.cpp. No actionable implementation or architectural findings remain from the rebase review.

2. Verification

  • 2.1 The rebased revision passed 273 selected Release tests on CI-Ubuntu-20. These included the renamed NXImageProcessing regressions, the four-bin RDF test, and the prebuilt pipeline parameter check. An independent enumeration matched all four RDF reference counts.
  • 2.2 After the include removal, the Release build and all six WriteINLFile tests passed on CI-Ubuntu-20. Clang-format 19.1.1 and git diff --check passed for the cleanup.
  • 2.3 This verification covers selected Release tests. Debug, disk-backed out-of-core execution, and the full test suite were outside this run.

3. Verdict

APPROVE. The requested fixes within this PR's scope are addressed.

@imikejackson

Copy link
Copy Markdown
Contributor Author

/azp run

@azure-pipelines

Copy link
Copy Markdown
Azure Pipelines:
Successfully started running 1 pipeline(s).

@imikejackson
imikejackson merged commit 010b34e into BlueQuartzSoftware:develop Nov 13, 2026
5 checks passed
@imikejackson
imikejackson deleted the topic/errata_1908_fixes branch November 13, 2026 20:16
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

BUG: Feature clustering RDF reference is not normalized to unit mass

2 participants