Repository navigation
BUG: Validate EBSD reader headers, scans and slice ranges - #1967
Closed
imikejackson wants to merge 5 commits into
Closed
imikejackson wants to merge 5 commits into
imikejackson wants to merge 5 commits into
Conversation
File-derived header values reached the preflight preview and the Image Geometry without checks: - The shared phase preview indexed the orientation operations with an unchecked symmetry, Laue group, IT or space group value, so an unsupported value crashed preflight (ReadAngData, ReadCtfData, ReadH5OimData, ReadH5EspritData, ReadH5OinaData, ReadChannel5Data, ReadH5Ebsd and ReadGrainMapper3D). Preflight now rejects it (-67600). - Header grid dimensions of zero or less reached unsigned shapes (-67601). - Header steps and resolutions that were negative, zero, NaN or infinite became geometry spacing (-67602). Esprit keeps its documented exact-zero default of 1. - ReadH5Ebsd checked an active sample-transformation axis only in execute (-67603). - ReadChannel5Data cell counts that overflow the reader's int32 arithmetic are rejected before the header is read (-67604, -67605). - ReadH5OimData and ReadH5EspritData check every selected scan header, not only the first. Each case has a regression test that failed on develop. The V&V deviation pages of ReadAngData, ReadCtfData and ReadH5OinaData record the new defects. Addresses BlueQuartzSoftware#1867 and BlueQuartzSoftware#1877 (EBSD reader header items) and BlueQuartzSoftware#1908 item L88. Signed-off-by: Michael Jackson <mike.jackson@bluequartz.net>
- ReadH5OimData and ReadH5EspritData sized the geometry and the ensemble arrays from the first selected scan only. A later scan with a smaller grid, other spacing, or another phase table read or wrote out of bounds in execute. Preflight and execute now require every selected scan to match the first one (-67606 to -67608, -67619, -67620), and execute checks the destination arrays before it copies (-67616 to -67618). - With Import Pattern Data on, the pattern dataset of each selected scan is checked for presence, unsigned 8-bit type, rank 3 and extents that match the header before EbsdLib reads it (-67609 to -67612). - ReadH5Ebsd accepted a slice range outside the file. For ANG-based files each missing slice was filled with zeros with no error; for CTF-based files execute failed with a generic error. Preflight now rejects the range (-67614). - ReadChannel5Data with compatible arrays or hexagonal alignment on required Euler fields that preflight did not check, so execute dereferenced a null field (-67615). - Remove stale comments in ReadH5EspritData.hpp that said bulk-I/O Results are not checked. Each defect has a regression test that failed before the fix. Addresses BlueQuartzSoftware#1867 (ReadH5EspritDataFilter#1, ReadH5OimDataFilter#2, ReadH5OimDataFilter#4, ReadH5EbsdFilter#6, ReadH5EbsdFilter#8, ReadChannel5DataFilter#2, RawPatterns check) and BlueQuartzSoftware#1899 item D19. Signed-off-by: Michael Jackson <mike.jackson@bluequartz.net>
- EbsdToH5Ebsd and ConvertHexGridToSquareGrid use one case-insensitive extension check, so .ANG and .Ctf files no longer pass preflight and then fail in execute. - ConvertHexGridToSquareGrid compared the temporary output path with the input path, so an empty Output Prefix in the input directory overwrote the input .ang file. The final output path is now checked in preflight and in execute (-44604). - ConvertHexGridToSquareGrid refreshes its cached preview for any changed file time, and no longer throws on a short '# ' header line. - ReadEnsembleInfo validates the phase count, phase groups, keys and names in preflight. A negative Number_Phases is rejected (-13003) instead of indexing an empty array. - ReadGrainMapper3D fills every mapped dataset type that preflight creates; other types than float32, int32 and uint8 stayed zero. - ReadGrainMapper3D rejects nonpositive or nonfinite LabDCT spacing (-67602), and nonfinite Extent, Center, VirtualShift or computed origin (-67621, -67622). - The ImportEdaxOIMData example pipeline reads the archived EdaxOIMData.h5 file instead of a file that is in no data archive, and is registered as a pipeline test. The Python example uses the current parameter names. Each defect has a regression test that failed before the fix. Addresses BlueQuartzSoftware#1867, BlueQuartzSoftware#1877 and BlueQuartzSoftware#1908 item L71. Closes BlueQuartzSoftware#1906. Signed-off-by: Michael Jackson <mike.jackson@bluequartz.net>
- ReadH5OinaData copied a NaN or infinite user Origin component into the Image Geometry. Preflight now rejects it (-9592). - ReadCtfData accepted an explicit ZCells of 0 in preflight and failed later in execute. Preflight now rejects it (-67601). A file without the ZCells key still imports as one slice, and a negative value still gives -19604. - Add V&V deviations ReadCtfDataFilter-D7 and ReadH5OinaDataFilter-D16 and update the V&V test inventories and provenance. Each defect has a regression test that failed before the fix. Addresses BlueQuartzSoftware#1908 item L87 and BlueQuartzSoftware#1917 item M43. Signed-off-by: Michael Jackson <mike.jackson@bluequartz.net>
- ReadH5Ebsd checks the sample-transformation axis with the same float normalization that execute uses, so tiny or huge finite axes are rejected in preflight (-67603). - ReadEnsembleInfo again fills the whole Crystal Structures and Phase Types arrays with Unknown before it writes the phase entries. - New ReadAngData tests set NCOLS_EVEN and NROWS to 0 and -1 and require -67601 in preflight and execute. - GrainMapper3D requires finite Extent, Center and VirtualShift values but checks the float32 range only on the computed origin and spacing. - The ReadH5OinaData Origin message names the file. - The ImportEdaxOIMData example writes its XDMF file again and keeps its numpy import; two V&V provenance sentences are corrected. Signed-off-by: Michael Jackson <mike.jackson@bluequartz.net>
|
Azure Pipelines: There may be pipelines that require an authorized user to comment /azp run to run. |
This was referenced Nov 12, 2026
Contributor
Author
|
Closed in favor of #1973, which merges this branch (with its history) together with the other EBSD fixes that need the next EbsdLib release (BlueQuartzSoftware/EbsdLib#86). The branch is kept. 🤖 Generated with Claude Code |
This was referenced Nov 13, 2026
imikejackson
added a commit
to imikejackson/simplnx
that referenced
this pull request
Nov 14, 2026
…Software#1967) Originally simplnx BlueQuartzSoftware#1967 (branch fix/ebsd-vv-filter-items), combined into this branch for the EbsdLib 3.1.4 update. Signed-off-by: Michael Jackson <mike.jackson@bluequartz.net>
imikejackson
added a commit
to imikejackson/simplnx
that referenced
this pull request
Nov 14, 2026
- The merge of BlueQuartzSoftware#1962 into the combined branch joined the last BlueQuartzSoftware#1967 test case and the first BlueQuartzSoftware#1962 test case without the closing brace between them, so ReadGrainMapper3DTest.cpp did not compile. Signed-off-by: Michael Jackson <mike.jackson@bluequartz.net>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
The EBSD reader filters and related OrientationAnalysis filters now check the values they read from a file before they use them. Before, a malformed header, a later scan that does not match the first scan, a short pattern dataset or a slice range outside the file gave a crash, wrong data with no error, or a late or unclear error. Most of these checks now run in preflight, and each one gives an error that names the file, the header key or dataset, and the value. Each defect has a regression test that uses a small synthetic file that the test writes. 53 files changed, +2840 / −189 lines.
1. Header values (symmetry, grid dimensions, steps)
src/Plugins/OrientationAnalysis/src/OrientationAnalysis/utilities/EbsdReaderUtilities.hpp— shared checks:ValidateHeaderDimension,ValidateHeaderSpacing,ValidateHeaderGrid, andGeneratePreflightPhaseInformation, which now returnsResult<>Filters/ReadAngDataFilter.cpp,ReadCtfDataFilter.cpp,ReadChannel5DataFilter.cpp,ReadH5OimDataFilter.cpp,ReadH5EspritDataFilter.cpp,ReadH5OinaDataFilter.cpp,ReadH5EbsdFilter.cpp,ReadGrainMapper3DFilter.cppThe phase preview looked up the orientation operations with a symmetry, Laue group, IT number or space group from the file, and it did not check that value. An unsupported value read out of bounds or dereferenced null in preflight. Preflight now rejects the value (-67600). GrainMapper3D accepts space groups 1 to 230.
Grid dimensions of zero or less are rejected before they become unsigned shapes (-67601). Steps and resolutions that are negative, zero, NaN or infinite are rejected before they become geometry spacing (-67602). Esprit keeps its documented behavior of using 1 for a step of exactly zero. Channel5 cell counts that overflow the reader's int32 arithmetic are rejected before EbsdLib reads the header (-67604, -67605). An explicit CTF
ZCellsof 0 is rejected; a file without the key still imports one slice. A NaN or infinite H5OINA Origin component is rejected (-9592).2. Multi-scan OIM and Esprit files and pattern data
src/Plugins/OrientationAnalysis/src/OrientationAnalysis/utilities/IEbsdOemReader.hpp— execute-side checksFilters/ReadH5OimDataFilter.cpp,Filters/ReadH5EspritDataFilter.cpp— preflight checks of each selected scanPreflight sized the geometry and the ensemble arrays from the first selected scan only. A later scan with fewer columns or rows, or more phases, gave out-of-bounds reads and writes in execute. Preflight now reads the header of each selected scan and requires it to match the first scan in dimensions, spacing, phase count, material names and lattice constants (-67606 to -67608, -67619, -67620). When pattern import is on, the
Pattern(OIM) orRawPatterns(Esprit) dataset of each scan must exist, be uint8, have rank 3 and have the extents (X × Y, height, width) (-67609 to -67612). Execute repeats the checks that protect its copies, so a call without a matching preflight also fails safely (-67616 to -67618).3. H5EBSD slice range and sample transformation
Filters/ReadH5EbsdFilter.cppThe selected start and end slices must be inside the slice range stored in the file (-67614). Before, an ANG-based file filled each missing slice with zeros and gave no error, and a CTF-based file failed late with a generic error. When the recommended transformation is on, the sample-transformation axis must normalize, in the same float arithmetic that execute uses, to finite nonzero values (-67603).
4. Channel5 required fields
Filters/ReadChannel5DataFilter.cpp,Filters/Algorithms/ReadChannel5Data.cppWith Convert Data Arrays to be more compatible with DREAM3D on, a
.cpr/.crcfile without the Euler fields made execute dereference null pointers. Preflight and execute now report the missing field (-67615).5. Related OrientationAnalysis filters
Filters/ConvertHexGridToSquareGridFilter.cpp,Filters/Algorithms/ConvertHexGridToSquareGrid.cpp,Filters/Algorithms/EbsdToH5Ebsd.cpp— one shared normalized-extension helper (NormalizedExtension), so.ANGand.Ctfare accepted in preflight and in execute; the output path is checked against the input path (ValidateHexOutputPath, -44604), so an empty prefix in the input directory no longer overwrites the input.angfile; short#header lines no longer throw; a changed file timestamp refreshes the suggested output text.Filters/ReadEnsembleInfoFilter.cpp,Filters/Algorithms/ReadEnsembleInfo.cpp—ReadEnsembleFilevalidates the phase count, the phase groups and the crystal-structure and phase-type text in preflight (-13001 to -13008). A negativeNumber_Phasesno longer reaches the output shape.Filters/Algorithms/ReadGrainMapper3D.cpp,utilities/GrainMapper3DUtilities.cpp— execute fills arrays of every type that preflight creates (before, only float32, int32 and uint8 were filled and the others stayed 0). LabDCT spacing must be finite and positive. Extent, Center and VirtualShift must be finite, and the computed origin must fit float32 (-67621, -67622).6. Example pipeline
src/Plugins/OrientationAnalysis/pipelines/EBSD_File_Processing/ImportEdaxOIMData.d3dpipelineandwrapping/python/examples/pipelines/Incomplete/ImportEdaxOIMData.pyThe pipeline read
Data/Hikari_Ni_Sequence.h5, which is in no data archive. It now readsData/OrientationAnalysis/EdaxOIMData.h5, Scan 1, and runs as a registered pipeline test. The Python example uses the current parameter names.7. Documentation and V&V
docs/Coding_Knowledge_Base.mdextends the rule about header-derived indices and dimensions.Umbrella items fixed
Each row is one checklist entry in an umbrella issue. "§N" is the numbered section above. "Row" is the file and line that the entry links to; the item ID is given where the entry has one. Test names omit the
OrientationAnalysis::prefix.#1867 (C07: preflight and execute disagree)
ReadChannel5DataFilter#2(ReadChannel5Data.cpp:276)ReadChannel5Data: StructuralValidation required fieldsReadH5EspritDataFilter#1(ReadH5EspritData.cpp:25)ReadH5EspritData: StructuralValidationReadH5OimDataFilter#2(ReadH5OimDataFilter.cpp:131)ReadH5OimData: StructuralValidationReadH5OimDataFilter#4(ReadH5OimData.cpp:103)ReadH5OimData: StructuralValidation,ReadH5OimData: StructuralValidation valid patterns and stale destinationReadH5EspritData: StructuralValidation,ReadH5EspritData: StructuralValidation valid patterns and stale destinationReadGrainMapper3DFilter#1(ReadGrainMapper3D.cpp:443)ReadGrainMapper3D: EBSD regression mapped scalar typesReadGrainMapper3DFilter#7(GrainMapper3DUtilities.cpp:30)ReadGrainMapper3D: HeaderValidation symmetryReadH5EbsdFilter#8(ReadH5EbsdFilter.cpp:119)ReadH5Ebsd: StructuralValidation slice range(ANG cases)ReadH5EbsdFilter#6(ReadH5EbsdFilter.cpp:119)ReadH5Ebsd: StructuralValidation slice range(CTF cases)EbsdToH5EbsdFilter#1(EbsdToH5Ebsd.cpp:155).ANG/.Ctfpass preflight and fail at executeEbsdToH5Ebsd: EBSD regression suffixReadAngData: HeaderValidation symmetryReadCtfData: HeaderValidation symmetryReadH5OimData: HeaderValidation(symmetry)ReadH5EspritData: HeaderValidation(symmetry)ReadChannel5Data: HeaderValidation(LaueGroup)ReadH5OinaData: HeaderValidation symmetry,ReadH5OinaData: HeaderValidation optional Space GroupReadH5Ebsd: HeaderValidation(LaueGroup, HKL file; the ANG path calls the same check)ConvertHexGridToSquareGrid: EBSD regression suffix,ConvertHexGridToSquareGrid: CTF suffix is case insensitiveConvertHexGridToSquareGrid: EBSD regression cache older timestampReadEnsembleInfo: EBSD regression preflight validation#1877 (C18: file parsing and output path checks)
ConvertHexGridToSquareGridFilter#3(ConvertHexGridToSquareGrid.cpp:93).angConvertHexGridToSquareGrid: EBSD regression same fileConvertHexGridToSquareGridFilter#5(ConvertHexGridToSquareGrid.cpp:278)#header line throwsstd::out_of_rangeConvertHexGridToSquareGrid: EBSD regression short commentsReadGrainMapper3D: EBSD regression finite header(Spacing)ReadGrainMapper3D: EBSD regression finite header,ReadGrainMapper3D: Computed origin fits geometryAbsorptionCT)ReadH5EspritData: HeaderValidation(columns,rows)ReadH5EspritData: HeaderValidation(width,height)ReadH5EspritData: HeaderValidation(xstep,ystep)ReadH5OimData: HeaderValidation(width,height)ReadH5OimData: HeaderValidation(columns,rows)ReadH5OimDataFilter#5)ReadH5OimData: HeaderValidation(xstep,ystep)ReadChannel5Data: HeaderValidation,ReadChannel5Data: HeaderValidation count limits and nonfinite stepsGridDist)XSTEP: 1e999gives infinite spacingReadAngData: HeaderValidation stepsReadCtfData: HeaderValidation steps,ReadCtfDataFilter: Zero Step Preflight Error (-67602)ReadH5Ebsd: HeaderValidation(Max X Points,Max Y Points)ReadH5Ebsd: HeaderValidation(X Resolution,Y Resolution)ReadH5Ebsd: HeaderValidation(SampleTransformationAxis),ReadH5Ebsd: Float normalization axisNumber_Phasespasses the zero-only checkReadEnsembleInfo: EBSD regression preflight validation#1908, #1917 and #1899
Pipelines#2(ImportEdaxOIMData.d3dpipeline:35)ImportEdaxOIMDatapipeline test (now registered)ReadH5OinaDataFilter: Origin components must be finite (-9592),ReadH5OinaDataFilter: Finite Origin bounds acceptedReadAngData: HeaderValidation dimensionsZCells 0fails only at executeReadCtfDataFilter: Zero ZCells preflight rejected (-67601)ReadH5EspritData.hppRelated entries that this PR does not fix
ReadH5OimDataFilter#3, scan stacking order): fixed by BUG: Fix 21 filter defects ported from 'SIMPL' and 'DREAM3D' #1915.ReadH5EbsdFilter#2: already fixed by BUG: Fix 18 filter defects introduced in the 6.5.171 port #1900.CreateEnsembleInfoFilter#1, and the ReadH5Ebsd entries at ReadH5EbsdFilter.cpp:214 (later-slice phases) and :119 (unrotated preflight metadata), and ConvertHexGridToSquareGrid.cpp:210 and :309 (text precision).ReadH5OimDataFilter#2/ReadH5EspritDataFilter#1andEbsdToH5EbsdFilter#1/ConvertHexGridToSquareGridFilter.cpp:143.Test plan
OrientationAnalysis::(EbsdToH5Ebsd|ConvertHexGridToSquareGrid|ReadEnsembleInfo|ReadGrainMapper3D|Read(AngData|CtfData|H5Ebsd|H5EspritData|H5OimData|H5OinaData|Channel5Data))|ImportEdaxOIMData.SIMPLNX_TEST_ALGORITHM_PATH=1): the same selection passes (121/121).-L "^ooc$" -LE benchmark): no new failures. The threenxrunner_ooc_*script tests also fail ondevelop(their scripts are missing).NXVtkLibTestsfailed only in a sandbox without an OpenGL context and passes when run normally.Related Issue
Closes #1906
Closes #1882
Related to #1867
Related to #1877
Related to #1908
Related to #1917
Related to #1899