Repository navigation
BUG: Fix #1908 wrong-data items L73, L74, L77, L78 and L79 - #1981
Open
imikejackson wants to merge 10 commits into
Open
imikejackson wants to merge 10 commits into
imikejackson wants to merge 10 commits into
Conversation
Remove out-of-plane kernel offsets on single-slice images before the empty-kernel checks. Previously, an Annulus with only Z offsets bypassed those checks and reduced no image values. Preserve each operation's existing empty-kernel result and leave 3D kernels unchanged. Add independent copy-through and zero-output tests for all seven grayscale filters on both algorithm paths, and for the bounded composite pipeline. Part of BlueQuartzSoftware#1908 (item L73) Signed-off-by: Michael Jackson <mike.jackson@bluequartz.net>
Legacy DREAM3D shift files omit the top section's zero row. Importing the 116 legacy rows as cumulative shifts for 117 sections skips the first row and applies the remaining shifts to the wrong sections. Replace the align_sections_input.tar.gz registration with align_sections_input_v2.tar.gz, which prepends the missing zero row. The v1 archive is retired from this registration and remains immutable. Update both example pipelines to use v2, import all 117 rows, document the row-to-section mapping, and test generated legacy and corrected files. Part of BlueQuartzSoftware#1908 (item L78) Signed-off-by: Michael Jackson <mike.jackson@bluequartz.net>
The seed triangle moves to the origin and contributes a zero row to the least-squares system. Small patches could therefore fit three or seven coefficients with too few usable neighbor rows and return arbitrary curvature. Count finite, nonzero non-seed rows for the selected fit and reuse the existing invalid-patch NaN outputs when there are too few. Add analytic paraboloid and minimum-size regressions and document the patch requirements. Part of BlueQuartzSoftware#1908 (item L74) Signed-off-by: Michael Jackson <mike.jackson@bluequartz.net>
Reference-slice mode stored reference-relative cumulative offsets in both shift arrays. Align Sections (List) accumulated those offsets again when reading Relative input, producing incorrect alignment from row 2 onward. Store differences of consecutive integer cumulative shifts in both Direct and Scanline paths. Preserve cumulative shifts and non-reference behavior. Add a literal shift oracle and List round-trip regression, and clarify the Relative and Cumulative Shifts documentation. The existing output test now derives Relative Shifts from the exemplar's Cumulative Shifts, preserving the original archive and the other exact exemplar comparisons. Part of BlueQuartzSoftware#1908 (item L77) Signed-off-by: Michael Jackson <mike.jackson@bluequartz.net>
ReadImage compared decoded segment byte counts with page 0. Different component counts and sample types with the same byte width could pass and be copied using page 0's layout, producing incorrect pixel values. Add IImageIO::readPageLayouts returning only small page metadata structs. The TIFF implementation opens the file once, collects logical page directories once and reads their tags with one forward traversal. ValidatePageLayouts compares those layouts in memory, preserving TIFF's unsigned sample-format default and the existing byte-count and dimension checks. ReadImageFilter rejects mismatched layouts in preflight with error -50579. The regression cases assert this preflight error explicitly. Keep the execute check because ImportFijiMontageFilter calls ReadImage directly without ReadImageFilter preflight; removing that guard requires addressing this caller. Add generated TIFF regression cases and document the page layout requirement. Part of BlueQuartzSoftware#1908 (item L79) Signed-off-by: Michael Jackson <mike.jackson@bluequartz.net>
Read Image Stack reads only page 0 of each file. The page-layout check compared every page of a multi-page TIFF, so a stack file whose unread pages had a different layout failed with -50579, although only page 0 is decoded. Pass the number of pages the read uses to ValidatePageLayouts in preflight and execute, and add a single-page-read regression test. Related to BlueQuartzSoftware#1908 item L79 Signed-off-by: Michael Jackson <mike.jackson@bluequartz.net>
FeatureFaceCurvature queued one worker per feature face even when the workers accessed shared out-of-core stores. These stores do not support concurrent access, so separate triangle ranges could still race and lose output values. Use ParallelTaskAlgorithm's residency gate with all input and output arrays in one requireArraysInMemory call. Run workers synchronously when any store is out-of-core and retain the existing TBB task queue when all stores are resident. Add multi-feature analytic plane tests with real disk-backed store witnesses and a deterministic queue-decision regression. Remove the filter from the knowledge base's ungated list. Signed-off-by: Michael Jackson <mike.jackson@bluequartz.net>
The page-layout check compared every page of a multi-page TIFF with page 0, so a Z-cropped read failed with -50579 when a page outside the crop had a different layout, although that page is never decoded. ValidatePageLayouts now takes the first page and the number of pages that the read decodes and compares only those pages with page 0, which sets the output array layout. Preflight checks after the crop is resolved, and execute checks with the same range before any output is written. Document exactly which pages are checked and add Z-crop regression cases. Related to BlueQuartzSoftware#1908 item L79 Signed-off-by: Michael Jackson <mike.jackson@bluequartz.net>
- Add 2D out-of-plane kernel tests and doc notes for Opening and Closing by Reconstruction (Preserve Intensities on), which the shared MorphologyEngine change also fixes. - Add a 2D-kernel note to every grayscale morphology filter page. - Reuse the in-plane kernel helper in the binary composite pipeline. - Replace the raw TIFF sample-format code in PageMetadata with a format-neutral SampleFormat enum. - Add direct readPageLayouts tests, a test that reaches the execute-time page-layout check, and a curvature test that detects lost writes. - Fix doc wording, test tags and comments; drop unneeded LoadPlugins(). Signed-off-by: Michael Jackson <mike.jackson@bluequartz.net>
The page-layout check rejects a TIFF whose pages differ in component count or data type in preflight with -50579, before the per-page byte count check can return -2007 in execute. Update the mixed RGB/grayscale test to expect -50579 from preflight and execute with no output array, and describe both errors on the Read Image page. Part of BlueQuartzSoftware#1908 (item L79) Signed-off-by: Michael Jackson <mike.jackson@bluequartz.net>
|
Azure Pipelines: There may be pipelines that require an authorized user to comment /azp run to run. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Fixes five wrong-data defects from the #1908 sweep (items L73, L74, L77, L78 and L79), and a related out-of-core race in Feature Face Curvature. Each fix has regression tests with independent expected values and updates the filter documentation that states the changed behavior. 36 files changed, +1507 / −70 lines.
Companion PR: BlueQuartzSoftware/SimplnxOoc#44 (private repository). It updates one out-of-core test that expects the old reference-slice Relative Shifts (L77). Merge the two together; without it, that one SimplnxOoc test fails in the out-of-core build.
Test data: the L78 fix uses a new data archive,
align_sections_input_v2.tar.gz, which is already published on theData_Archiverelease. The originalalign_sections_input.tar.gzis unchanged and is no longer referenced.1. Grayscale morphology on 2D images (L73)
src/simplnx/Utilities/ImageProcessing/MorphologyEngine.hpp:detail::MakeInPlaneMorphologyKernel()src/Plugins/NXImageProcessing/docs/*ImageFilter.md: 9 morphology pagesOn an image with Z = 1, an out-of-plane kernel (for example an Annulus around Z) used to read outside the slice and write wrong values. The engine now drops every kernel offset with dz ≠ 0 when Z = 1. If no offset is left, the existing empty-kernel behavior applies: Dilate, Erode, Opening and Closing copy the input, and Morphological Gradient and the top-hats output zero. The fix applies to every grayscale route (Direct, Scanline, gradient and the composite pipelines). It therefore also fixes Opening and Closing by Reconstruction with Preserve Intensities on, which used to output INT32_MIN or INT32_MAX in this case. All 9 filter pages describe the 2D behavior.
2. Legacy align-sections shift file (L78)
src/Plugins/SimplnxCore/CMakeLists.txt: downloadsalign_sections_input_v2.tar.gzsrc/Plugins/SimplnxCore/pipelines/Align_Sections_Examples/*.d3dpipelinesrc/Plugins/SimplnxCore/docs/AlignSectionsListFilter.mdThe 6.5.171 shift file
Alignment_By_Misorientation_Shifts.txthas 116 rows for 117 sections. It is missing the zero row of the top section, so Align Sections (List) applied every shift to the wrong section. The v2 archive adds the row116 117 0 0 0 0, and the example pipelines read 117 rows from it. The documentation says that row 0 is the top section, that it must be a zero row for either input type, and that rows go from the top section to the bottom section. The unit test writes its own legacy file in code and does not use the archive.3. Feature Face Curvature: too few triangles for the fit (L74)
src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/CalculateTriangleGroupCurvatures.cppThe seed triangle moves to the origin, so its row in the least-squares system is zero. A small patch could therefore fit 7 (with face normals) or 3 coefficients from fewer usable rows and return arbitrary curvatures. The algorithm now counts the usable neighbor rows (finite, not all zero). When there are fewer rows than coefficients, all requested outputs of that triangle are NaN, the same as for other invalid patches.
4. Feature Face Curvature: out-of-core race
src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/FeatureFaceCurvature.cppdocs/Coding_Knowledge_Base.mdThe algorithm ran one TBB task per feature face, and the tasks read and wrote shared stores. Out-of-core stores do not support concurrent writes, so values could be lost. A
ParallelTaskAlgorithmnow callsrequireArraysInMemory()with all 13 arrays that the workers use. The tasks run in parallel only when all of these arrays are in memory; otherwise they run one after another. The knowledge base no longer lists Feature Face Curvature as a filter without this gate.5. Align Sections (Feature Centroid): relative shifts in reference-slice mode (L77)
src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/AlignSectionsFeatureCentroid.cppsrc/Plugins/SimplnxCore/docs/AlignSectionsFeatureCentroidFilter.mdWith Use Reference Slice on, the Relative Shifts array held the shift relative to the reference slice, which is the same as the Cumulative Shifts. Read back into Align Sections (List) as relative input, these shifts were added up a second time. Relative Shifts now hold the slice-to-slice difference of the cumulative shifts in both the in-core and the out-of-core path. The first row has no slice above it, so its relative shift equals its cumulative shift. Cumulative Shifts and the consecutive-slice mode do not change. The test computes the expected relative shifts from the existing exemplar's cumulative shifts, so the exemplar archive does not change.
6. Read Image: multi-page TIFFs with different page layouts (L79)
src/simplnx/Utilities/ImageIO/IImageIO.hpp,TiffImageIO.hpp/.cpp: newIImageIO::readPageLayouts()andPageMetadatasrc/Plugins/NXImageProcessing/src/NXImageProcessing/Filters/Algorithms/ReadImage.cpp:ReadImage::ValidatePageLayouts()src/Plugins/NXImageProcessing/src/NXImageProcessing/Filters/ReadImageFilter.cppThe reader took the component count and data type from page 0 and read every page with that layout. A page with a different layout gave wrong values.
readPageLayouts()reads the size, component count, bits per sample and a format-neutralSampleFormatof every page in one pass. The default implementation inIImageIOreturns one page.ValidatePageLayouts()compares only the pages that the read uses (page 0 plus the pages inside a Z crop; one page for a single-page read) and returns error-50579, which names the page and both layouts. Preflight runs the check. Execute runs it again, which protects callers that use the algorithm directly for multi-page reads without the filter's preflight, such as Import Fiji Montage.The page-layout check runs before the per-page
-2007checks from #1915. A file with an 8-bit RGB page followed by an 8-bit grayscale page now fails in preflight with-50579, not in execute with-2007. The existing testMultiPage_TIFF_RGB_Then_Grayscale_Rejected_Before_Copyexpects this, and the Read Image page describes both errors. Pages with different dimensions still fail with-2007while the filter reads them.Test plan
NXImageProcessing::MorphologyEngine2D out-of-plane kernel tests (L73), and the 2D passthrough tests of Opening and Closing by Reconstruction.SimplnxCore::AlignSectionsListFilter: Legacy cumulative file requires zero row(L78).SimplnxCore::FeatureFaceCurvatureFilter: Patch solvability(L74): analytic paraboloid patches at and below the minimum size.SimplnxCore::FeatureFaceCurvatureFilter: multi-feature planesand... with real OOC stores: outputs are pre-filled with a sentinel, so a lost write fails.SimplnxCore::AlignSectionsFeatureCentroidFilter: Reference slice relative shifts round trip(L77): literal expected shifts, and Align Sections (List) gives the same result from relative and cumulative input.NXImageProcessing::ReadImageFilter: TIFF_PageLayout,TIFF_PageLayout_SinglePageOnly,TIFF_PageLayout_ZCrop,NXImageProcessing::ReadImage: TIFF_PageLayout rejected in execute, and theTiffImageIO:: readPageLayouts ...tests (L79).Nonzero reference top slice,Empty slices) check the new slice-to-slice Relative Shifts.MorphologyEngine|Grayscale|MorphologicalGradient|TopHat|Reconstruction|AlignSections|FeatureFaceCurvature|ReadImage|ImageIO|Tiff|ImportFijiMontage) and the Small IN100 and align-sections pipeline tests pass, run one at a time.ooclabel (1053 tests) pass.Related Issue
Related to #1908
Related to #1815