Skip to content

fix (display): leaked runtime when display loads follow each other quickly - #4017

Open
emilioheredia-source wants to merge 1 commit into
ControlSystemStudio:masterfrom
emilioheredia-source:display/serialize-display-loads
Open

emilioheredia-source wants to merge 1 commit into
ControlSystemStudio:masterfrom
emilioheredia-source:display/serialize-display-loads

Conversation

@emilioheredia-source

@emilioheredia-source emilioheredia-source commented Nov 12, 2026 •

Copy link
Copy Markdown

Summary

DisplayRuntimeInstance.loadDisplayFile() disposes the shown display, shows "Loading ...", and loads the new display in a background job, but nothing kept track of that job. When a second load starts before the first one is done, for example with a double click on "back", both jobs represent their display and start its runtime:

  • One runtime keeps running, with its PVs connected and its scripts and rules active, until Phoebus exits.
  • Whichever load finishes last is shown, which can be a different display than the navigation history says.

Closing the display while it loads also gave a "Cannot load model" error dialog after the display was gone.

The load sequence stays as it is: the shown display is disposed and "Loading ..." is shown right away, the file is loaded in the background, and neither the loads nor the UI thread wait for each other. What's new:

  • The display instance keeps its load Job. A new load, or closing the display, cancels it (Job.cancel(), as in the logbook search).
  • A cancelled job doesn't represent its display or start its runtime, and its errors are only logged. The check before representing runs on the UI thread, where the cancellation happens.
  • A display runtime that is starting can't be stopped (DisplayRuntime.stop() skips it while Starting). If the display is disposed during that start, the UI thread only disposes the representation and the load job stops the runtime and disposes the model once the start is done. The UI thread doesn't wait for the start, which matters for PV types that connect while they are created.

Testing

  • DisplayLoadSequenceUI (TestFX), 5 tests. Two loads in a row, a replaced load that fails, closing a display while it loads, and a newer load while the previous runtime is still starting (with a test PV type whose creation blocks) fail on master and pass with this PR. One test checks that the shown display's PVs are released and "Loading ..." is shown before the new widgets, on master and with this PR.
  • Real application, history A, B (400 PVs), C, "back" twice 20 ms apart: master left B's runtime running in 5 runs out of 5 and once showed B; with this PR, in 13 runs, A is shown and every runtime that started was stopped. Without the hand-over for a starting runtime, 1 run in 5 still left B running. "back"/"forward" 5 times 15 ms apart: master ended on the wrong display with 3 extra runtimes; with this PR, the right display and one runtime. Closing the tab, or Phoebus, right after "back": no errors, every runtime stopped.
  • Restoring 15 displays in 3 windows at startup: same time as master.

Not changed here, the same as on master:

  • A load that fails just before a newer load starts can still clear the newer load's display info or close its display.
  • The "replace" target of an Open Display action doesn't go through loadDisplayFile() and has a similar window between representing and starting.

Checklist

  • Testing:
    • The feature has automated tests
    • Tests were run

The tests follow the *UI naming of the ui-tests profile. That profile currently runs no tests (failsafe 2.22.0 can't discover JUnit 5 tests with the current JUnit platform, also for the existing UI tests in core/ui), so they were run with surefire under Xvfb.

@georgweiss

georgweiss commented Nov 12, 2026 •

Copy link
Copy Markdown
Collaborator

@emilioheredia-source , this looks very interesting. Would you mind testing a case where a large number (>10) of non-trivial displays are open, maybe across multiple windows? The test should just shut down Phoebus and restart to recreate the displays as they were when shutting down (default Phoebus behavior).

Reason I ask is to figure out how user experience would be affected, e.g. seemingly longer time to restore the view when launching Phoebus.

@emilioheredia-source

Copy link
Copy Markdown
Author

Thanks, that was a good question. The test found a real slowdown, which is fixed now.

Setup: 15 displays from the examples folder (10_scripts, 01_main, 11_datasources, 12_formulas, plots_xy, plots_image, structure_tabs, monitors_table, monitors_symbols, ...), 5 tabs in each of 3 windows, saved as the memento. Phoebus is started and restores them. I measured the time from launching Phoebus until the last display logged "Done with representing model", alternating master and the PR, on Xvfb with software rendering.

All 15 displays represented First display
master (9 runs) 3.12 – 3.20 s, one run 3.90 s about 2.7 s
PR as first pushed (2 runs) 3.60 – 3.77 s about 3.2 s
PR now (4 runs) 3.13 – 3.23 s about 2.7 s

The first version was about 0.4 s slower. Each load went to the UI thread twice: once to dispose the shown display before loading, then again to represent the new one. At startup, the first layout pass of the restored windows ran in between, so every display waited for it.

The new commit disposes the shown display and represents the new one in a single UI step, after the new display has been loaded, as master does for the representation. With that, restoring takes the same time as on master. One visible difference when navigating: the shown display now remains until the new one is ready, instead of being replaced by a "Loading ..." message right away. The first display of a tab still shows the message.

In case it helps to judge the impact: we came across the leak while adding the mouse back/forward buttons for our operators, who navigate regularly with Back and Forward in EDM-style windows. A double click on the toolbar's Back button is enough to leave the skipped display's PVs connected until Phoebus exits.

@shroffk

shroffk commented Nov 13, 2026 •

Copy link
Copy Markdown
Member

Thanks @emilioheredia-source
This is an important leak you have discovered and thanks for the insight into the problem.

I believe we have moved the file loading off the UI thread due to various performance issues discovered in BOY. This is particularly true if you have a bad script...

That having being said the resource leak you point out is a serious bug.

then loads the new one in a background job. A load that is still in its job is not tracked anywhere until its display is represented. When a second load starts in that time, nothing disposes the first one, and its job still represents the display and starts its runtime.

Based on your finding the key issue here is that the job is not tracked properly. Instead of getting rid of the background loading could be make it into a proper cancellable job ( using CompletableFuture maybe? ) and a proper tracking of it. This would address the leak while preserving the UX ( visual indication that new file is being loading by removing the old one and being more performant in situation where multiple files are being loaded )

I remember with BOY when the existing screen stayed visisble... instead of waiting users were inclined to keep pressing the button more and more - out of frustration - making matters worse.

Again, we are quite glad you have discovered this

@emilioheredia-source
emilioheredia-source force-pushed the display/serialize-display-loads branch from caef31a to e49520c Compare November 15, 2026 16:37
@emilioheredia-source

Copy link
Copy Markdown
Author

Thanks @shroffk, and sorry. You're right, and I should have asked before changing this. Reading the file did stay in the background job, but I changed when the old display goes away: it stayed on screen, still running, until the new one was ready, and loads in the same tab waited for each other. That throws away exactly the immediate feedback you describe from BOY. I made that change to win back the startup time Georg asked about, instead of stepping back to a smaller fix.

The new version keeps the original flow as it was: the old display is removed and "Loading" is shown on the click, and the file is read in a background job, without loads waiting for each other and without the UI thread waiting for a load. The addition is what you suggested: the tab keeps track of its current load job (a Phoebus Job, as in the logbook search) and cancels it when a new load starts or the tab is closed. A cancelled job doesn't draw its display or start its runtime, and doesn't report errors.

Restoring 15 displays in 3 windows takes the same time as on master. With history A, B, C and a double click on "back", master left B's runtime running or showed B; with this version A is shown and every runtime that started was stopped.

One part needed a decision, and I'd like your opinion on it: a display runtime that is still starting can't be stopped (DisplayRuntime skips the stop in its Starting state). If the display is disposed during that start, the UI thread now only disposes the representation, and the load job stops the runtime and disposes the model once the start is done, so the UI never waits for slow PV connections. The details are in the updated description.

…ckly

loadDisplayFile() disposes the shown display, shows "Loading ...", and
loads the new display in a background job, but nothing kept track of
that job. When a second load started before the first one was done, for
example on a double click on "back", both jobs represented their display
and started its runtime: one runtime kept running with its PVs until
exit, or the wrong display was shown. Closing the display while it
loaded gave a "Cannot load model" error.

Keep the load sequence as it is and track the load job: a new load or
closing the display cancels it. A cancelled job doesn't represent its
display or start its runtime, and its errors are only logged. The check
before representing runs on the UI thread, where the cancellation
happens.

A display runtime that is starting can't be stopped. When the display
is disposed during that start, the UI thread only disposes the
representation, and the load job stops the runtime and disposes the
model once the start is done. The UI thread never waits for it.
@emilioheredia-source
emilioheredia-source force-pushed the display/serialize-display-loads branch from e49520c to 93dee88 Compare November 15, 2026 16:39
@sonarqubecloud

Copy link
Copy Markdown

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants