Skip to content

BUG: Validate EBSD reader headers, scans and slice ranges - #1967

Closed
imikejackson wants to merge 5 commits into
BlueQuartzSoftware:developfrom
imikejackson:fix/ebsd-vv-filter-items
Closed

imikejackson wants to merge 5 commits into
BlueQuartzSoftware:developfrom
imikejackson:fix/ebsd-vv-filter-items

Conversation

@imikejackson

@imikejackson imikejackson commented Nov 12, 2026 •

Copy link
Copy Markdown
Contributor

Summary

The EBSD reader filters and related OrientationAnalysis filters now check the values they read from a file before they use them. Before, a malformed header, a later scan that does not match the first scan, a short pattern dataset or a slice range outside the file gave a crash, wrong data with no error, or a late or unclear error. Most of these checks now run in preflight, and each one gives an error that names the file, the header key or dataset, and the value. Each defect has a regression test that uses a small synthetic file that the test writes. 53 files changed, +2840 / −189 lines.

Blocked — do not merge yet. Merge this PR only after an EbsdLib release that contains BlueQuartzSoftware/EbsdLib#72 is available to simplnx. EbsdLib#72 fixes the root cause of the ANG .h5ebsd missing-slice defect in H5AngReader/H5AngVolumeReader. This PR adds the preflight slice-range check in simplnx. The code in this PR builds and passes its tests with the current EbsdLib (3.1.3).

Merge order: #1913 merges first; this branch is then rebased onto develop. #1915 and #1962 also change IEbsdOemReader.hpp and GrainMapper3DUtilities.cpp; whichever PR merges second rebases.

Note for the reviewing agent: This change has already passed an alternating review (Codex and Claude, in turns) and a review-until-clean loop, which ended with no Critical or Important findings. Do not do a general review of each filter that this PR touches. Review what changed, and check that each change fixes the defect that it is reported to fix (the section Umbrella items fixed below lists each checklist entry from #1867, #1877, #1908, #1917 and #1899 with its fix and its regression test), that its regression test exercises that defect, and that the change does not go beyond that defect.

1. Header values (symmetry, grid dimensions, steps)

  • src/Plugins/OrientationAnalysis/src/OrientationAnalysis/utilities/EbsdReaderUtilities.hpp — shared checks: ValidateHeaderDimension, ValidateHeaderSpacing, ValidateHeaderGrid, and GeneratePreflightPhaseInformation, which now returns Result<>
  • Filters/ReadAngDataFilter.cpp, ReadCtfDataFilter.cpp, ReadChannel5DataFilter.cpp, ReadH5OimDataFilter.cpp, ReadH5EspritDataFilter.cpp, ReadH5OinaDataFilter.cpp, ReadH5EbsdFilter.cpp, ReadGrainMapper3DFilter.cpp

The phase preview looked up the orientation operations with a symmetry, Laue group, IT number or space group from the file, and it did not check that value. An unsupported value read out of bounds or dereferenced null in preflight. Preflight now rejects the value (-67600). GrainMapper3D accepts space groups 1 to 230.

Grid dimensions of zero or less are rejected before they become unsigned shapes (-67601). Steps and resolutions that are negative, zero, NaN or infinite are rejected before they become geometry spacing (-67602). Esprit keeps its documented behavior of using 1 for a step of exactly zero. Channel5 cell counts that overflow the reader's int32 arithmetic are rejected before EbsdLib reads the header (-67604, -67605). An explicit CTF ZCells of 0 is rejected; a file without the key still imports one slice. A NaN or infinite H5OINA Origin component is rejected (-9592).

2. Multi-scan OIM and Esprit files and pattern data

  • src/Plugins/OrientationAnalysis/src/OrientationAnalysis/utilities/IEbsdOemReader.hpp — execute-side checks
  • Filters/ReadH5OimDataFilter.cpp, Filters/ReadH5EspritDataFilter.cpp — preflight checks of each selected scan

Preflight sized the geometry and the ensemble arrays from the first selected scan only. A later scan with fewer columns or rows, or more phases, gave out-of-bounds reads and writes in execute. Preflight now reads the header of each selected scan and requires it to match the first scan in dimensions, spacing, phase count, material names and lattice constants (-67606 to -67608, -67619, -67620). When pattern import is on, the Pattern (OIM) or RawPatterns (Esprit) dataset of each scan must exist, be uint8, have rank 3 and have the extents (X × Y, height, width) (-67609 to -67612). Execute repeats the checks that protect its copies, so a call without a matching preflight also fails safely (-67616 to -67618).

3. H5EBSD slice range and sample transformation

  • Filters/ReadH5EbsdFilter.cpp

The selected start and end slices must be inside the slice range stored in the file (-67614). Before, an ANG-based file filled each missing slice with zeros and gave no error, and a CTF-based file failed late with a generic error. When the recommended transformation is on, the sample-transformation axis must normalize, in the same float arithmetic that execute uses, to finite nonzero values (-67603).

4. Channel5 required fields

  • Filters/ReadChannel5DataFilter.cpp, Filters/Algorithms/ReadChannel5Data.cpp

With Convert Data Arrays to be more compatible with DREAM3D on, a .cpr/.crc file without the Euler fields made execute dereference null pointers. Preflight and execute now report the missing field (-67615).

5. Related OrientationAnalysis filters

  • Filters/ConvertHexGridToSquareGridFilter.cpp, Filters/Algorithms/ConvertHexGridToSquareGrid.cpp, Filters/Algorithms/EbsdToH5Ebsd.cpp — one shared normalized-extension helper (NormalizedExtension), so .ANG and .Ctf are accepted in preflight and in execute; the output path is checked against the input path (ValidateHexOutputPath, -44604), so an empty prefix in the input directory no longer overwrites the input .ang file; short # header lines no longer throw; a changed file timestamp refreshes the suggested output text.
  • Filters/ReadEnsembleInfoFilter.cpp, Filters/Algorithms/ReadEnsembleInfo.cpp — ReadEnsembleFile validates the phase count, the phase groups and the crystal-structure and phase-type text in preflight (-13001 to -13008). A negative Number_Phases no longer reaches the output shape.
  • Filters/Algorithms/ReadGrainMapper3D.cpp, utilities/GrainMapper3DUtilities.cpp — execute fills arrays of every type that preflight creates (before, only float32, int32 and uint8 were filled and the others stayed 0). LabDCT spacing must be finite and positive. Extent, Center and VirtualShift must be finite, and the computed origin must fit float32 (-67621, -67622).

6. Example pipeline

  • src/Plugins/OrientationAnalysis/pipelines/EBSD_File_Processing/ImportEdaxOIMData.d3dpipeline and wrapping/python/examples/pipelines/Incomplete/ImportEdaxOIMData.py

The pipeline read Data/Hikari_Ni_Sequence.h5, which is in no data archive. It now reads Data/OrientationAnalysis/EdaxOIMData.h5, Scan 1, and runs as a registered pipeline test. The Python example uses the current parameter names.

7. Documentation and V&V

  • Each changed filter's documentation page has a short sentence about its new error.
  • The V&V deviation pages of ReadAngData (D6), ReadCtfData (D6, D7) and ReadH5OinaData (D15, D16) record the defects and their fixes; their reports and provenance pages list the new tests.
  • docs/Coding_Knowledge_Base.md extends the rule about header-derived indices and dimensions.

Umbrella items fixed

Each row is one checklist entry in an umbrella issue. "§N" is the numbered section above. "Row" is the file and line that the entry links to; the item ID is given where the entry has one. Test names omit the OrientationAnalysis:: prefix.

#1867 (C07: preflight and execute disagree)

Item Defect Fix Regression test
ReadChannel5DataFilter#2 (ReadChannel5Data.cpp:276) Missing Euler/Phase fields with Convert on: null dereference §4, -67615 ReadChannel5Data: StructuralValidation required fields
ReadH5EspritDataFilter#1 (ReadH5EspritData.cpp:25) Later scan smaller than the first: heap out-of-bounds read §2, -67606 to -67608, -67619, -67620 ReadH5EspritData: StructuralValidation
ReadH5OimDataFilter#2 (ReadH5OimDataFilter.cpp:131) Geometry and ensemble sized from the first scan only §2, same codes ReadH5OimData: StructuralValidation
ReadH5OimDataFilter#4 (ReadH5OimData.cpp:103) Pattern dataset size not checked §2, -67609 to -67612, -67616 to -67618 ReadH5OimData: StructuralValidation, ReadH5OimData: StructuralValidation valid patterns and stale destination
ReadH5EspritData.cpp:109 RawPatterns type, rank and extents not checked §2, -67609 to -67612, -67616 to -67618 ReadH5EspritData: StructuralValidation, ReadH5EspritData: StructuralValidation valid patterns and stale destination
ReadGrainMapper3DFilter#1 (ReadGrainMapper3D.cpp:443) Arrays of types other than float32, int32 and uint8 stay 0 §5 ReadGrainMapper3D: EBSD regression mapped scalar types
ReadGrainMapper3DFilter#7 (GrainMapper3DUtilities.cpp:30) SpaceGroup outside 1 to 230 throws or maps to cubic §1, -67600 (#1962 has the same check and message) ReadGrainMapper3D: HeaderValidation symmetry
ReadH5EbsdFilter#8 (ReadH5EbsdFilter.cpp:119) ANG end slice past the file fills zeros with no error §3, -67614 (root cause: EbsdLib#72) ReadH5Ebsd: StructuralValidation slice range (ANG cases)
ReadH5EbsdFilter#6 (ReadH5EbsdFilter.cpp:119) CTF out-of-range slice fails late with -50003 §3, -67614 ReadH5Ebsd: StructuralValidation slice range (CTF cases)
EbsdToH5EbsdFilter#1 (EbsdToH5Ebsd.cpp:155) .ANG/.Ctf pass preflight and fail at execute §5 EbsdToH5Ebsd: EBSD regression suffix
ReadAngDataFilter.cpp:123 Symmetry not checked before the preview lookup §1, -67600 ReadAngData: HeaderValidation symmetry
ReadCtfDataFilter.cpp:149 Laue Group not checked before the preview lookup §1, -67600 ReadCtfData: HeaderValidation symmetry
ReadH5OimDataFilter.cpp:148 Symmetry not checked before the preview lookup §1, -67600 ReadH5OimData: HeaderValidation (symmetry)
ReadH5EspritDataFilter.cpp:156 IT number not checked before the Space Group lookup §1, -67600 ReadH5EspritData: HeaderValidation (symmetry)
ReadChannel5DataFilter.cpp:147 Laue Group not checked before the preview lookup §1, -67600 ReadChannel5Data: HeaderValidation (LaueGroup)
ReadH5OinaDataFilter.cpp:331 Laue Group not checked before the preview lookup §1, -67600 ReadH5OinaData: HeaderValidation symmetry, ReadH5OinaData: HeaderValidation optional Space Group
ReadH5EbsdFilter.cpp:150 Symmetry not checked before the lookup (ANG and HKL) §1, -67600 ReadH5Ebsd: HeaderValidation (LaueGroup, HKL file; the ANG path calls the same check)
ConvertHexGridToSquareGridFilter.cpp:143 Extension case compared raw §5 ConvertHexGridToSquareGrid: EBSD regression suffix, ConvertHexGridToSquareGrid: CTF suffix is case insensitive
ConvertHexGridToSquareGridFilter.cpp:158 Older timestamp keeps stale suggested output text §5 ConvertHexGridToSquareGrid: EBSD regression cache older timestamp
ReadEnsembleInfoFilter.cpp:120 Phase keys and text checked only at execute §5, -13004 to -13008 ReadEnsembleInfo: EBSD regression preflight validation

#1877 (C18: file parsing and output path checks)

Item Defect Fix Regression test
ConvertHexGridToSquareGridFilter#3 (ConvertHexGridToSquareGrid.cpp:93) Empty prefix in the input directory overwrites the input .ang §5, -44604 ConvertHexGridToSquareGrid: EBSD regression same file
ConvertHexGridToSquareGridFilter#5 (ConvertHexGridToSquareGrid.cpp:278) # header line throws std::out_of_range §5 ConvertHexGridToSquareGrid: EBSD regression short comments
GrainMapper3DUtilities.cpp:174 LabDCT spacing not checked §5, -67602 ReadGrainMapper3D: EBSD regression finite header (Spacing)
GrainMapper3DUtilities.cpp:206 Nonfinite LabDCT Extent, Center or VirtualShift §5, -67621 ReadGrainMapper3D: EBSD regression finite header, ReadGrainMapper3D: Computed origin fits geometry
GrainMapper3DUtilities.cpp:267 Nonfinite AbsorptionCT Center or VirtualShift §5, -67621 same two tests (AbsorptionCT)
ReadH5EspritDataFilter.cpp:143 Nonpositive grid dimensions §1, -67601 ReadH5EspritData: HeaderValidation (columns, rows)
ReadH5EspritDataFilter.cpp:208 Negative pattern dimensions §1, -67601 ReadH5EspritData: HeaderValidation (width, height)
ReadH5EspritDataFilter.cpp:146 XSTEP/YSTEP of -1, NaN or infinity §1, -67602 ReadH5EspritData: HeaderValidation (xstep, ystep)
ReadH5OimDataFilter.cpp:200 Negative pattern dimensions §1, -67601 ReadH5OimData: HeaderValidation (width, height)
ReadH5OimDataFilter.cpp:138 Signed scan dimensions reach unsigned shapes §1, -67601 ReadH5OimData: HeaderValidation (columns, rows)
ReadH5OimDataFilter.cpp:141 (header part of ReadH5OimDataFilter#5) Step X/Y of infinity becomes spacing §1, -67602 ReadH5OimData: HeaderValidation (xstep, ystep)
ReadChannel5DataFilter.cpp:136 Signed CPR cell counts not checked §1, -67601, -67604, -67605 ReadChannel5Data: HeaderValidation, ReadChannel5Data: HeaderValidation count limits and nonfinite steps
ReadChannel5DataFilter.cpp:139 GridDistX/Y of -1, NaN or infinity §1, -67602 the same two tests (GridDist)
ReadAngDataFilter.cpp:119 XSTEP: 1e999 gives infinite spacing §1, -67602 ReadAngData: HeaderValidation steps
ReadCtfDataFilter.cpp:145 XStep/YStep of -1 or NaN §1, -67602 ReadCtfData: HeaderValidation steps, ReadCtfDataFilter: Zero Step Preflight Error (-67602)
ReadH5EbsdFilter.cpp:119 File X/Y dimensions not checked §1, -67601 ReadH5Ebsd: HeaderValidation (Max X Points, Max Y Points)
ReadH5EbsdFilter.cpp:124 Root X/Y Resolution of 0, -1, NaN or infinity §1, -67602 ReadH5Ebsd: HeaderValidation (X Resolution, Y Resolution)
ReadH5EbsdFilter.cpp:252 Invalid sample-transformation axis reaches preflight §3, -67603 ReadH5Ebsd: HeaderValidation (SampleTransformationAxis), ReadH5Ebsd: Float normalization axis
ReadEnsembleInfoFilter.cpp:120 Negative Number_Phases passes the zero-only check §5, -13003 ReadEnsembleInfo: EBSD regression preflight validation

#1908, #1917 and #1899

Issue Item Defect Fix Regression test
#1908 L71 Pipelines#2 (ImportEdaxOIMData.d3dpipeline:35) Pipeline reads a file that is in no data archive §6 the ImportEdaxOIMData pipeline test (now registered)
#1908 L87 (ReadH5OinaDataFilter.cpp:326) Nonfinite user Origin reaches the geometry §1, -9592 ReadH5OinaDataFilter: Origin components must be finite (-9592), ReadH5OinaDataFilter: Finite Origin bounds accepted
#1908 L88 (ReadAngDataFilter.cpp:116) Nonpositive grid dimensions reach unsigned shapes §1, -67601 ReadAngData: HeaderValidation dimensions
#1917 M43 (ReadCtfDataFilter.cpp:131) Explicit ZCells 0 fails only at execute §1, -67601 ReadCtfDataFilter: Zero ZCells preflight rejected (-67601)
#1899 D19, ReadH5EspritData part only Stale comments say bulk-I/O Results are not checked comment fix in ReadH5EspritData.hpp none (comment only)

Related entries that this PR does not fix

Test plan

  • Each defect has a regression test that failed before its fix.
  • macOS Release in-core (DREAM3D-NX build): the tests of every changed filter and the ImportEdaxOIMData pipeline test pass (118/118): OrientationAnalysis::(EbsdToH5Ebsd|ConvertHexGridToSquareGrid|ReadEnsembleInfo|ReadGrainMapper3D|Read(AngData|CtfData|H5Ebsd|H5EspritData|H5OimData|H5OinaData|Channel5Data))|ImportEdaxOIMData.
  • macOS Release out-of-core (SIMPLNX_TEST_ALGORITHM_PATH=1): the same selection passes (121/121).
  • Full out-of-core label (-L "^ooc$" -LE benchmark): no new failures. The three nxrunner_ooc_* script tests also fail on develop (their scripts are missing). NXVtkLibTests failed only in a sandbox without an OpenGL context and passes when run normally.
  • clang-format 19.1.1 is clean on every changed C++ file.
  • CI passes on all platforms.

Related Issue

Closes #1906
Closes #1882
Related to #1867
Related to #1877
Related to #1908
Related to #1917
Related to #1899

File-derived header values reached the preflight preview and the Image
Geometry without checks:

- The shared phase preview indexed the orientation operations with an
  unchecked symmetry, Laue group, IT or space group value, so an
  unsupported value crashed preflight (ReadAngData, ReadCtfData,
  ReadH5OimData, ReadH5EspritData, ReadH5OinaData, ReadChannel5Data,
  ReadH5Ebsd and ReadGrainMapper3D). Preflight now rejects it (-67600).
- Header grid dimensions of zero or less reached unsigned shapes (-67601).
- Header steps and resolutions that were negative, zero, NaN or infinite
  became geometry spacing (-67602). Esprit keeps its documented
  exact-zero default of 1.
- ReadH5Ebsd checked an active sample-transformation axis only in
  execute (-67603).
- ReadChannel5Data cell counts that overflow the reader's int32
  arithmetic are rejected before the header is read (-67604, -67605).
- ReadH5OimData and ReadH5EspritData check every selected scan header,
  not only the first.

Each case has a regression test that failed on develop. The V&V
deviation pages of ReadAngData, ReadCtfData and ReadH5OinaData record
the new defects.

Addresses BlueQuartzSoftware#1867 and BlueQuartzSoftware#1877 (EBSD reader header items) and BlueQuartzSoftware#1908 item L88.

Signed-off-by: Michael Jackson <mike.jackson@bluequartz.net>
- ReadH5OimData and ReadH5EspritData sized the geometry and the ensemble
  arrays from the first selected scan only. A later scan with a smaller
  grid, other spacing, or another phase table read or wrote out of
  bounds in execute. Preflight and execute now require every selected
  scan to match the first one (-67606 to -67608, -67619, -67620), and
  execute checks the destination arrays before it copies (-67616 to
  -67618).
- With Import Pattern Data on, the pattern dataset of each selected scan
  is checked for presence, unsigned 8-bit type, rank 3 and extents that
  match the header before EbsdLib reads it (-67609 to -67612).
- ReadH5Ebsd accepted a slice range outside the file. For ANG-based
  files each missing slice was filled with zeros with no error; for
  CTF-based files execute failed with a generic error. Preflight now
  rejects the range (-67614).
- ReadChannel5Data with compatible arrays or hexagonal alignment on
  required Euler fields that preflight did not check, so execute
  dereferenced a null field (-67615).
- Remove stale comments in ReadH5EspritData.hpp that said bulk-I/O
  Results are not checked.

Each defect has a regression test that failed before the fix.

Addresses BlueQuartzSoftware#1867 (ReadH5EspritDataFilter#1, ReadH5OimDataFilter#2,
ReadH5OimDataFilter#4, ReadH5EbsdFilter#6, ReadH5EbsdFilter#8,
ReadChannel5DataFilter#2, RawPatterns check) and BlueQuartzSoftware#1899 item D19.

Signed-off-by: Michael Jackson <mike.jackson@bluequartz.net>
- EbsdToH5Ebsd and ConvertHexGridToSquareGrid use one case-insensitive
  extension check, so .ANG and .Ctf files no longer pass preflight and
  then fail in execute.
- ConvertHexGridToSquareGrid compared the temporary output path with
  the input path, so an empty Output Prefix in the input directory
  overwrote the input .ang file. The final output path is now checked
  in preflight and in execute (-44604).
- ConvertHexGridToSquareGrid refreshes its cached preview for any
  changed file time, and no longer throws on a short '# ' header line.
- ReadEnsembleInfo validates the phase count, phase groups, keys and
  names in preflight. A negative Number_Phases is rejected (-13003)
  instead of indexing an empty array.
- ReadGrainMapper3D fills every mapped dataset type that preflight
  creates; other types than float32, int32 and uint8 stayed zero.
- ReadGrainMapper3D rejects nonpositive or nonfinite LabDCT spacing
  (-67602), and nonfinite Extent, Center, VirtualShift or computed
  origin (-67621, -67622).
- The ImportEdaxOIMData example pipeline reads the archived
  EdaxOIMData.h5 file instead of a file that is in no data archive,
  and is registered as a pipeline test. The Python example uses the
  current parameter names.

Each defect has a regression test that failed before the fix.

Addresses BlueQuartzSoftware#1867, BlueQuartzSoftware#1877 and BlueQuartzSoftware#1908 item L71. Closes BlueQuartzSoftware#1906.

Signed-off-by: Michael Jackson <mike.jackson@bluequartz.net>
- ReadH5OinaData copied a NaN or infinite user Origin component into
  the Image Geometry. Preflight now rejects it (-9592).
- ReadCtfData accepted an explicit ZCells of 0 in preflight and failed
  later in execute. Preflight now rejects it (-67601). A file without
  the ZCells key still imports as one slice, and a negative value
  still gives -19604.
- Add V&V deviations ReadCtfDataFilter-D7 and ReadH5OinaDataFilter-D16
  and update the V&V test inventories and provenance.

Each defect has a regression test that failed before the fix.

Addresses BlueQuartzSoftware#1908 item L87 and BlueQuartzSoftware#1917 item M43.

Signed-off-by: Michael Jackson <mike.jackson@bluequartz.net>
- ReadH5Ebsd checks the sample-transformation axis with the same float
  normalization that execute uses, so tiny or huge finite axes are
  rejected in preflight (-67603).
- ReadEnsembleInfo again fills the whole Crystal Structures and Phase
  Types arrays with Unknown before it writes the phase entries.
- New ReadAngData tests set NCOLS_EVEN and NROWS to 0 and -1 and require
  -67601 in preflight and execute.
- GrainMapper3D requires finite Extent, Center and VirtualShift values
  but checks the float32 range only on the computed origin and spacing.
- The ReadH5OinaData Origin message names the file.
- The ImportEdaxOIMData example writes its XDMF file again and keeps its
  numpy import; two V&V provenance sentences are corrected.

Signed-off-by: Michael Jackson <mike.jackson@bluequartz.net>
@imikejackson imikejackson self-assigned this Nov 12, 2026
@azure-pipelines

Copy link
Copy Markdown
Azure Pipelines:
There may be pipelines that require an authorized user to comment /azp run to run.

@imikejackson

Copy link
Copy Markdown
Contributor Author

Closed in favor of #1973, which merges this branch (with its history) together with the other EBSD fixes that need the next EbsdLib release (BlueQuartzSoftware/EbsdLib#86). The branch is kept.

🤖 Generated with Claude Code

imikejackson added a commit to imikejackson/simplnx that referenced this pull request Nov 14, 2026
…Software#1967)

Originally simplnx BlueQuartzSoftware#1967 (branch fix/ebsd-vv-filter-items), combined into this branch for the EbsdLib 3.1.4 update.

Signed-off-by: Michael Jackson <mike.jackson@bluequartz.net>
imikejackson added a commit to imikejackson/simplnx that referenced this pull request Nov 14, 2026
- The merge of BlueQuartzSoftware#1962 into the combined branch joined the last BlueQuartzSoftware#1967 test case
  and the first BlueQuartzSoftware#1962 test case without the closing brace between them, so
  ReadGrainMapper3DTest.cpp did not compile.

Signed-off-by: Michael Jackson <mike.jackson@bluequartz.net>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

1 participant