Repository navigation
Verify FMI results against the variables of the reference - #409
Merged
Merged
Conversation
#399 gave FMPy the variables of the reference for FMI 2.0, but not the fmpy-fmi3-cs-cvode simulator of #400. That FMPy wrote only the outputs of the model. When the verifying omc started to fail on a variable that is not in the result (OpenModelica #17888), master-fmi3-cs-cvode lost 7241 verified models in one night. Give fmpy-fmi3-cs-cvode the same --output-variables. An FMI 3.0 FMU of OpenModelica can have an array as one variable with dimensions, x for x[1] to x[3], and FMPy writes all of its elements into one CSV cell, separated by spaces. So also give FMPy the names of the arrays, and after the simulation, give each element of an array its own column, named from the dimensions in modelDescription.xml. Assisted-by: Claude Opus 5.5
Reference results that Dymola wrote have variables that are not in the model: CPUtime, EventCounter, Time, _dummy and _derdummy (the state of a model without states), NonlinearSystems.* (statistics of the solvers) and _GlobalScope.* (constants of packages). OpenModelica never writes them. Since the verifying omc fails on a variable that is not in the result (OpenModelica #17888), these variables made models of ClaRa and ThermofluidStream fail in all lanes: 280 to 950 models per lane in the runs of 2026-11-09. Remove them from the variables of the reference, and ignore them in the variables that diffSimulationResults reports. Assisted-by: Claude Opus 5.5
readSimulationResultVars(openmodelicaStyle=true) gives the names of a reference in OpenModelica style: no spaces, and der(a.b) for a.der(b). The names of comparisonSignals.txt were used as written. The MSL references write stack.cell[1, 1].y, so the variableFilter did not match stack.cell[1,1].y and the simulation did not write it, and FMPy did not find it in the FMU. Give the names of comparisonSignals.txt the same style. Assisted-by: Claude Opus 5.5
The CSV references of Buildings write the header without quotes, as time,booRep.y[1,1]. A CSV reader splits booRep.y[1,1] into booRep.y[1 and 1], so these variables were never compared: in the runs of 2026-11-09, 25 to 30 models of Buildings per lane failed this way. Add the configuration field referenceFilePreprocessor. It names a function that writes a copy of each reference file, and the models are compared against the copy. quoteCsvHeader joins the header fields that a comma inside [] or () split apart and puts each name in quotes. Use it for the CSV references of Buildings. Assisted-by: Claude Opus 5.5
sjoelund
enabled auto-merge
November 10, 2026 06:52
sjoelund
disabled auto-merge
November 10, 2026 08:05
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The runs of 2026-11-09 lost many verified models: master-fmi3-cs-cvode 7241, each FMI 2.0 lane 500 to 1000, and master 504. No model changed. The omc that verifies (the nightly in /usr/bin, "Verified using" in the reports) is now dev-1807. It has OpenModelica/OpenModelica#17888: diffSimulationResults fails on a variable of the reference that is not in the result. Before, it did not compare such a variable. In all the regressed rows, the only failures are such missing variables.
The commits remove the causes that are in the library testing:
--output-variables, so FMPy wrote only the outputs. An FMI 3.0 FMU of OpenModelica can have an array as one variable, and FMPy writes its elements into one CSV cell, separated by spaces. FMPy now also gets the names of the arrays, and each element gets its own column, from the dimensions in modelDescription.xml._dummy,_derdummy,NonlinearSystems.*and_GlobalScope.*. They made ClaRa and ThermofluidStream models fail in all lanes.stack.cell[1, 1].y. The variableFilter did not matchstack.cell[1,1].y, so the simulation did not write it, and FMPy did not find it in the FMU.referenceFilePreprocessor. The CSV references of Buildings write the header without quotes (time,booRep.y[1,1]), so a CSV reader splits the names at the comma.quoteCsvHeaderwrites a copy with the names in quotes, and the models are compared against it. Buildings uses it. This is necessary before the host gets an omc with [API] Read result files through libomc_result OpenModelica#17962: its CSV reader refuses such a file, and then the model shows as verified with no variable compared.The FMPy lanes also need OpenModelica/OpenModelica#17971 (find
a[1, 2]in a CSV result that hasa[1,2]) and OpenModelica/OpenModelica#17972 (FMPy writes Booleans as True and False).Tested locally: the FMI job of test.py with all five simulators, with omc dev.1836 and a libomc_result that has #17971 and #17972, on ClaRa TestGenerateGrid, Modelica 4.1.0 CCCV_Stack and Buildings BooleanVectorReplicator. /mnt/ReferenceFiles is not available locally, so the Buildings reference was made from a result, with a header without quotes. All 15 runs verify. Before, TestGenerateGrid failed in all 5, CCCV_Stack failed in the 2 FMPy runs, and BooleanVectorReplicator compared no variable.
Assisted-by: Claude Opus 5.5