Skip to content

BUG: Fix #1908 wrong-data items L73, L74, L77, L78 and L79 - #1981

Open
imikejackson wants to merge 10 commits into
BlueQuartzSoftware:developfrom
imikejackson:topic/issue1908_batch4
Open

imikejackson wants to merge 10 commits into
BlueQuartzSoftware:developfrom
imikejackson:topic/issue1908_batch4

Conversation

@imikejackson

@imikejackson imikejackson commented Nov 14, 2026 •

Copy link
Copy Markdown
Contributor

Summary

Fixes five wrong-data defects from the #1908 sweep (items L73, L74, L77, L78 and L79), and a related out-of-core race in Feature Face Curvature. Each fix has regression tests with independent expected values and updates the filter documentation that states the changed behavior. 36 files changed, +1507 / −70 lines.

Companion PR: BlueQuartzSoftware/SimplnxOoc#44 (private repository). It updates one out-of-core test that expects the old reference-slice Relative Shifts (L77). Merge the two together; without it, that one SimplnxOoc test fails in the out-of-core build.

Test data: the L78 fix uses a new data archive, align_sections_input_v2.tar.gz, which is already published on the Data_Archive release. The original align_sections_input.tar.gz is unchanged and is no longer referenced.

1. Grayscale morphology on 2D images (L73)

  • src/simplnx/Utilities/ImageProcessing/MorphologyEngine.hpp: detail::MakeInPlaneMorphologyKernel()
  • src/Plugins/NXImageProcessing/docs/*ImageFilter.md: 9 morphology pages

On an image with Z = 1, an out-of-plane kernel (for example an Annulus around Z) used to read outside the slice and write wrong values. The engine now drops every kernel offset with dz ≠ 0 when Z = 1. If no offset is left, the existing empty-kernel behavior applies: Dilate, Erode, Opening and Closing copy the input, and Morphological Gradient and the top-hats output zero. The fix applies to every grayscale route (Direct, Scanline, gradient and the composite pipelines). It therefore also fixes Opening and Closing by Reconstruction with Preserve Intensities on, which used to output INT32_MIN or INT32_MAX in this case. All 9 filter pages describe the 2D behavior.

2. Legacy align-sections shift file (L78)

  • src/Plugins/SimplnxCore/CMakeLists.txt: downloads align_sections_input_v2.tar.gz
  • src/Plugins/SimplnxCore/pipelines/Align_Sections_Examples/*.d3dpipeline
  • src/Plugins/SimplnxCore/docs/AlignSectionsListFilter.md

The 6.5.171 shift file Alignment_By_Misorientation_Shifts.txt has 116 rows for 117 sections. It is missing the zero row of the top section, so Align Sections (List) applied every shift to the wrong section. The v2 archive adds the row 116 117 0 0 0 0, and the example pipelines read 117 rows from it. The documentation says that row 0 is the top section, that it must be a zero row for either input type, and that rows go from the top section to the bottom section. The unit test writes its own legacy file in code and does not use the archive.

3. Feature Face Curvature: too few triangles for the fit (L74)

  • src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/CalculateTriangleGroupCurvatures.cpp

The seed triangle moves to the origin, so its row in the least-squares system is zero. A small patch could therefore fit 7 (with face normals) or 3 coefficients from fewer usable rows and return arbitrary curvatures. The algorithm now counts the usable neighbor rows (finite, not all zero). When there are fewer rows than coefficients, all requested outputs of that triangle are NaN, the same as for other invalid patches.

4. Feature Face Curvature: out-of-core race

  • src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/FeatureFaceCurvature.cpp
  • docs/Coding_Knowledge_Base.md

The algorithm ran one TBB task per feature face, and the tasks read and wrote shared stores. Out-of-core stores do not support concurrent writes, so values could be lost. A ParallelTaskAlgorithm now calls requireArraysInMemory() with all 13 arrays that the workers use. The tasks run in parallel only when all of these arrays are in memory; otherwise they run one after another. The knowledge base no longer lists Feature Face Curvature as a filter without this gate.

5. Align Sections (Feature Centroid): relative shifts in reference-slice mode (L77)

  • src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/AlignSectionsFeatureCentroid.cpp
  • src/Plugins/SimplnxCore/docs/AlignSectionsFeatureCentroidFilter.md

With Use Reference Slice on, the Relative Shifts array held the shift relative to the reference slice, which is the same as the Cumulative Shifts. Read back into Align Sections (List) as relative input, these shifts were added up a second time. Relative Shifts now hold the slice-to-slice difference of the cumulative shifts in both the in-core and the out-of-core path. The first row has no slice above it, so its relative shift equals its cumulative shift. Cumulative Shifts and the consecutive-slice mode do not change. The test computes the expected relative shifts from the existing exemplar's cumulative shifts, so the exemplar archive does not change.

6. Read Image: multi-page TIFFs with different page layouts (L79)

  • src/simplnx/Utilities/ImageIO/IImageIO.hpp, TiffImageIO.hpp/.cpp: new IImageIO::readPageLayouts() and PageMetadata
  • src/Plugins/NXImageProcessing/src/NXImageProcessing/Filters/Algorithms/ReadImage.cpp: ReadImage::ValidatePageLayouts()
  • src/Plugins/NXImageProcessing/src/NXImageProcessing/Filters/ReadImageFilter.cpp

The reader took the component count and data type from page 0 and read every page with that layout. A page with a different layout gave wrong values. readPageLayouts() reads the size, component count, bits per sample and a format-neutral SampleFormat of every page in one pass. The default implementation in IImageIO returns one page. ValidatePageLayouts() compares only the pages that the read uses (page 0 plus the pages inside a Z crop; one page for a single-page read) and returns error -50579, which names the page and both layouts. Preflight runs the check. Execute runs it again, which protects callers that use the algorithm directly for multi-page reads without the filter's preflight, such as Import Fiji Montage.

The page-layout check runs before the per-page -2007 checks from #1915. A file with an 8-bit RGB page followed by an 8-bit grayscale page now fails in preflight with -50579, not in execute with -2007. The existing test MultiPage_TIFF_RGB_Then_Grayscale_Rejected_Before_Copy expects this, and the Read Image page describes both errors. Pages with different dimensions still fail with -2007 while the filter reads them.

Test plan

  • New regression tests with independent expected values:
    • NXImageProcessing::MorphologyEngine 2D out-of-plane kernel tests (L73), and the 2D passthrough tests of Opening and Closing by Reconstruction.
    • SimplnxCore::AlignSectionsListFilter: Legacy cumulative file requires zero row (L78).
    • SimplnxCore::FeatureFaceCurvatureFilter: Patch solvability (L74): analytic paraboloid patches at and below the minimum size.
    • SimplnxCore::FeatureFaceCurvatureFilter: multi-feature planes and ... with real OOC stores: outputs are pre-filled with a sentinel, so a lost write fails.
    • SimplnxCore::AlignSectionsFeatureCentroidFilter: Reference slice relative shifts round trip (L77): literal expected shifts, and Align Sections (List) gives the same result from relative and cumulative input.
    • NXImageProcessing::ReadImageFilter: TIFF_PageLayout, TIFF_PageLayout_SinglePageOnly, TIFF_PageLayout_ZCrop, NXImageProcessing::ReadImage: TIFF_PageLayout rejected in execute, and the TiffImageIO:: readPageLayouts ... tests (L79).
  • The L73, L74, L77 and L79 tests, the Reconstruction passthrough tests, the execute-time page-layout test and the sentinel curvature test were run against the code before their fix and failed.
  • The existing reference-slice tests (Nonzero reference top slice, Empty slices) check the new slice-to-slice Relative Shifts.
  • macOS arm64 Release, in-core: affected tests (MorphologyEngine|Grayscale|MorphologicalGradient|TopHat|Reconstruction|AlignSections|FeatureFaceCurvature|ReadImage|ImageIO|Tiff|ImportFijiMontage) and the Small IN100 and align-sections pipeline tests pass, run one at a time.
  • macOS arm64 Release, out-of-core (DREAM3D-NX build with SimplnxOoc and the companion PR): the same affected tests and the full ooc label (1053 tests) pass.
  • CI passes on all platforms.

Related Issue

Related to #1908
Related to #1815

Remove out-of-plane kernel offsets on single-slice images before the
empty-kernel checks. Previously, an Annulus with only Z offsets bypassed
those checks and reduced no image values. Preserve each operation's
existing empty-kernel result and leave 3D kernels unchanged.

Add independent copy-through and zero-output tests for all seven grayscale
filters on both algorithm paths, and for the bounded composite pipeline.

Part of BlueQuartzSoftware#1908 (item L73)

Signed-off-by: Michael Jackson <mike.jackson@bluequartz.net>
Legacy DREAM3D shift files omit the top section's zero row. Importing the
116 legacy rows as cumulative shifts for 117 sections skips the first row
and applies the remaining shifts to the wrong sections.

Replace the align_sections_input.tar.gz registration with
align_sections_input_v2.tar.gz, which prepends the missing zero row.
The v1 archive is retired from this registration and remains immutable.
Update both example pipelines to use v2, import all 117 rows, document the
row-to-section mapping, and test generated legacy and corrected files.

Part of BlueQuartzSoftware#1908 (item L78)

Signed-off-by: Michael Jackson <mike.jackson@bluequartz.net>
The seed triangle moves to the origin and contributes a zero row to the
least-squares system. Small patches could therefore fit three or seven
coefficients with too few usable neighbor rows and return arbitrary curvature.

Count finite, nonzero non-seed rows for the selected fit and reuse the
existing invalid-patch NaN outputs when there are too few. Add analytic
paraboloid and minimum-size regressions and document the patch requirements.

Part of BlueQuartzSoftware#1908 (item L74)

Signed-off-by: Michael Jackson <mike.jackson@bluequartz.net>
Reference-slice mode stored reference-relative cumulative offsets in both
shift arrays. Align Sections (List) accumulated those offsets again when
reading Relative input, producing incorrect alignment from row 2 onward.

Store differences of consecutive integer cumulative shifts in both Direct
and Scanline paths. Preserve cumulative shifts and non-reference behavior.
Add a literal shift oracle and List round-trip regression, and clarify the
Relative and Cumulative Shifts documentation. The existing output test now
derives Relative Shifts from the exemplar's Cumulative Shifts, preserving
the original archive and the other exact exemplar comparisons.

Part of BlueQuartzSoftware#1908 (item L77)

Signed-off-by: Michael Jackson <mike.jackson@bluequartz.net>
ReadImage compared decoded segment byte counts with page 0. Different
component counts and sample types with the same byte width could pass and
be copied using page 0's layout, producing incorrect pixel values.

Add IImageIO::readPageLayouts returning only small page metadata structs.
The TIFF implementation opens the file once, collects logical page directories
once and reads their tags with one forward traversal. ValidatePageLayouts
compares those layouts in memory, preserving TIFF's unsigned sample-format
default and the existing byte-count and dimension checks.

ReadImageFilter rejects mismatched layouts in preflight with error -50579.
The regression cases assert this preflight error explicitly. Keep the execute
check because ImportFijiMontageFilter calls ReadImage directly without
ReadImageFilter preflight; removing that guard requires addressing this caller.

Add generated TIFF regression cases and document the page layout requirement.

Part of BlueQuartzSoftware#1908 (item L79)

Signed-off-by: Michael Jackson <mike.jackson@bluequartz.net>
Read Image Stack reads only page 0 of each file. The page-layout check
compared every page of a multi-page TIFF, so a stack file whose unread
pages had a different layout failed with -50579, although only page 0 is
decoded. Pass the number of pages the read uses to ValidatePageLayouts in
preflight and execute, and add a single-page-read regression test.

Related to BlueQuartzSoftware#1908 item L79

Signed-off-by: Michael Jackson <mike.jackson@bluequartz.net>
FeatureFaceCurvature queued one worker per feature face even when the
workers accessed shared out-of-core stores. These stores do not support
concurrent access, so separate triangle ranges could still race and
lose output values.

Use ParallelTaskAlgorithm's residency gate with all input and output
arrays in one requireArraysInMemory call. Run workers synchronously
when any store is out-of-core and retain the existing TBB task queue
when all stores are resident. Add multi-feature analytic plane tests
with real disk-backed store witnesses and a deterministic queue-decision
regression. Remove the filter from the knowledge base's ungated list.

Signed-off-by: Michael Jackson <mike.jackson@bluequartz.net>
The page-layout check compared every page of a multi-page TIFF with page 0,
so a Z-cropped read failed with -50579 when a page outside the crop had a
different layout, although that page is never decoded. ValidatePageLayouts
now takes the first page and the number of pages that the read decodes and
compares only those pages with page 0, which sets the output array layout.
Preflight checks after the crop is resolved, and execute checks with the
same range before any output is written. Document exactly which pages are
checked and add Z-crop regression cases.

Related to BlueQuartzSoftware#1908 item L79

Signed-off-by: Michael Jackson <mike.jackson@bluequartz.net>
- Add 2D out-of-plane kernel tests and doc notes for Opening and Closing
  by Reconstruction (Preserve Intensities on), which the shared
  MorphologyEngine change also fixes.
- Add a 2D-kernel note to every grayscale morphology filter page.
- Reuse the in-plane kernel helper in the binary composite pipeline.
- Replace the raw TIFF sample-format code in PageMetadata with a
  format-neutral SampleFormat enum.
- Add direct readPageLayouts tests, a test that reaches the
  execute-time page-layout check, and a curvature test that detects
  lost writes.
- Fix doc wording, test tags and comments; drop unneeded LoadPlugins().

Signed-off-by: Michael Jackson <mike.jackson@bluequartz.net>
The page-layout check rejects a TIFF whose pages differ in component
count or data type in preflight with -50579, before the per-page byte
count check can return -2007 in execute. Update the mixed RGB/grayscale
test to expect -50579 from preflight and execute with no output array,
and describe both errors on the Read Image page.

Part of BlueQuartzSoftware#1908 (item L79)

Signed-off-by: Michael Jackson <mike.jackson@bluequartz.net>
@imikejackson imikejackson self-assigned this Nov 14, 2026
@azure-pipelines

Copy link
Copy Markdown
Azure Pipelines:
There may be pipelines that require an authorized user to comment /azp run to run.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant