Repository navigation
odyssey, wgpu: compose the GPU display only on change, and run that path headless in CI - #698
Open
jamesbraid wants to merge 4 commits into
Open
jamesbraid wants to merge 4 commits into
jamesbraid wants to merge 4 commits into
Conversation
…hanged
With the GPU raster unit (raster = "wgpu") the display still composed
every frame, so an idle V12 Tezro at the IRIX greeter kept the display
thread busy: 14-16% of a host core and about 23 billion user instructions
a minute, against 0.1% and 4.3 million with the CPU raster unit. The CPU
path stopped doing that when stores to board memory started setting a flag
the display takes before each frame, but the GPU unit draws, copies,
uploads and clears with no CPU store, so the flag never saw its work and
both GPU paths composed unconditionally to stay correct.
The GPU unit now sets the flag through the mirror it already holds after
each write to the memory the scan-out reads: the batch submit in flush,
the copy submit, the dirty-block and region uploads, written words and the
memory clear. The uploads mark again because the display may take a CPU
store's mark before its bytes reach the GPU memory it composes from;
without that mark it would show the old bytes until the next draw. The
scan-out runs on the same queue as the raster batches, so a mark set
after a submit either lands before the display takes the flag, and the
compose sees the batch, or after, and the next frame composes again.
Reader::update no longer composes every frame when the GPU scan-out is
present, and update_on_gpu (the windowed wgpu renderer's path) keeps its
last buffer and what it was composed from: an unchanged frame returns that
buffer unchanged, and the display presents it only at the idle refresh, as
on the CPU path. As on the CPU path, a frame composed for a mark is
followed by one more (the settling flag): a mark is relaxed and can be
seen before the data it marks, so the next frame catches it. The marks are
one relaxed store per submit or upload, not per store; nothing is added to
VramPtr::store.
Measured on an idle V12 Tezro at the greeter (2 CPUs, jitv2, headless,
raster = "wgpu" on an RTX 4090 over Vulkan, 60 s idle then 28 s of 40
pointer moves; four runs of each build, alternating base, branch, branch,
base, base, branch, branch, base; user instructions of the display thread
by perf stat; base is main without this commit):
- idle: base 23.02-23.03 G instructions, 14-16% of a core; branch
4.30 M, 0.1% in all four runs
- pointer moving: base 10.55-10.75 G, 15-16%; branch 253-259 M, 0.5%
- control, raster = "cpu", one boot of each build: 11.070 G in the
pointer window for both, idle 4.31 M against 4.30 M
- the greeter screenshot is byte-identical in every run, CPU and GPU
Only the headless path ran: no display server was available, so the
windowed update_on_gpu path is covered by a test, not a boot, and
nothing is measured on arm64 (the added cost there is a relaxed store per
GPU submit).
Tests: every_store_to_gpu_memory_marks_the_display, on a real GPU, takes
the flag, then checks that a submitted batch, a dirty-block upload after
the display took a store's mark, a region upload, written words and a
memory clear each set it again, and that an empty flush does not.
Removing any one of those five marks fails it. A copy is covered by the
display tests with VPRO_GPU=1 ("a copy" in the change table), where
removing the copy's mark fails; the_gpu_scanout_composes_only_a_changed_frame
drives update_on_gpu and fails if it ignores the flag, the settling frame,
or composes always.
The idle display test no longer skips on the GPU unit.
Whether the VPro display skips unchanged frames could only be seen indirectly, as the display thread's share of a host core: 23 billion user instructions a minute at an idle IRIX greeter before the GPU raster unit marked its writes, 4.3 million after. A test or a nightly check that wants to catch that coming back needs the display to say what it did, not a perf counter on a named thread. The reader now counts frames composed, those composed only to settle the frame after a store's mark, and those left unchanged. The display counts frames presented, and how many of those came from a GPU buffer. The display thread times each frame. The monitor shows all of it with `vpro frames`, and `vpro frames reset` zeroes it, so a harness can read a window of idle time and expect zero composed frames and one present in six. The reader's counts are plain integers under the reader lock the frame already holds. The display's are relaxed atomics only the display thread adds to, and the frame timing reuses the clock read that paces the thread. Measured on an idle V12 Tezro greeter (x86-64, release, ci mode, display thread instructions:u over 60 s, runs alternated C D D C): 4.58 million without the counts, 5.16 million with them, about 160 instructions a frame at 60 Hz. With 40 pointer moves over 28 s, 253.0 and 253.2 million without against 253.4 and 253.3 million with. arm64 was not measured: the added work is two relaxed adds, a compare-exchange loop and one clock read a frame, never per pixel or per store.
The wgpu window is the only renderer that takes frames still on the board's GPU, so only a window ran the VPro's GPU scan-out (update_on_gpu) and the present after it. ci mode with no window installs no renderer at all and always composed on the CPU path. None of that path's cost or freshness could be tested in CI or measured on a host with no display. On a workstation with two RTX 4090s and no X server, the change that stopped the GPU path composing every frame could be checked only by a reader-level test, never through the display thread. Presenter gains an offscreen target: a texture the window's size in place of the surface. Pipelines, layers, the unpack kernel and the pass are the same. Only acquiring and presenting the surface texture are left out, and after each submit the previous frame's submission is waited on, so no more than two frames are in flight, as the window's frame latency allows. Nothing is read back. WgpuRenderer keeps its window's handles in an optional WindowLink, and ui::draw_offscreen builds the same renderer with no window. main.rs installs it in ci mode without a window when renderer = "wgpu", so the board takes the window's whole path: gpu_frames_wanted, update_on_gpu, due, present_gpu, present_gpu_frame and Presenter::present. The default renderer is gl, so ci runs that do not ask for wgpu are unchanged. A V12 Tezro booted this way to the IRIX greeter on an RTX 4090 reports, through `vpro frames`, 0 frames composed and 650 presented from the GPU in an idle minute, and 40 composed for 40 pointer moves. With the changed-only check disabled, the same minute composes and presents all 3900 frames, and the display thread runs 2.96 billion instructions against 0.26 billion. The presenter test reads the offscreen texture back and checks the frame and the status bar where the window would have them. The display test runs Odyssey::frame with the offscreen renderer on a GPU board and checks, through the frame counts, that idle frames are neither composed nor presented outside the idle refresh, that a register change or a new mode composes one frame and a draw two, and that every present comes from the GPU. Both pass on an RTX 4090 and on lavapipe with no GPU. Each count the display test asserts fails when the behaviour it guards is broken: the skip, the settle frame, the renderer taking GPU frames, the idle refresh.
No workflow builds the wgpu feature, so the VPro GPU raster unit, its
scan-out and the wgpu presenter run in CI only as code that compiles
under other features, and their tests skip without a GPU. A display that
went back to composing every frame on the GPU path cost 23 billion
display-thread instructions a minute at an idle greeter. It would land
green, and nothing short of a window on a real GPU would show it.
Hosted runners have no GPU, but lavapipe, Mesa's software Vulkan driver,
opens as a wgpu adapter with no code change. In an ubuntu:24.04
container with mesa-vulkan-drivers 25.2.8 and no other driver, the 41
tests this job selects pass in 28 s on four threads. The workflow itself
has not run on GitHub.
The new workflow installs mesa-vulkan-drivers on ubuntu-24.04 and runs
`cargo ci-gpu`. The alias builds the wgpu feature for the library and
the binaries, so the ci-mode wiring in main.rs is compiled too, and its
nextest profile picks the wgpu layer and the VPro display tests. Those
assert the display's frame counts rather than compare pictures. With
VPRO_GPU=1 a board that cannot open the adapter fails, so a run without
a working lavapipe goes red. The profile runs four tests at a time:
lavapipe refused devices ("parent device is lost") when dozens opened at
once on a many-core host. The VPro golden scenes stay out: two differ
from their goldens by one colour step on the GPU unit. Pull requests run
it when they touch the wgpu layer, the VPro board, the window or this
configuration, since it is a build of its own.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
With
raster = "wgpu"and the wgpu window, the VPro display composes and presents on the GPU. Until now only an open window ran that path. Nothing in CI could catch it going back to composing every frame, and at an idle IRIX greeter that cost 23 billion display-thread instructions a minute. Hosted runners have no GPU and no display. Screenshots force the CPU path, so they can't see it either.This makes the window's path run with no window and has the display report what it did:
update_on_gpukeeps the last frame when nothing it reads has changed.vpro frames [reset]reports frames composed, composed only to settle a store, left unchanged and presented (and how many from the GPU), plus frame time.Presentercan draw into an offscreen texture instead of a surface, with the same pipelines and pass.WgpuRendererkeeps its window handles in an optionalWindowLink. ci mode withrenderer = "wgpu"installs the same renderer offscreen, soOdyssey::framerunsupdate_on_gpu,present_gpu,present_gpu_frameandPresenter::present. Only acquiring and presenting the surface texture are left out. The default renderer is gl, so other ci runs are unchanged.wgpu.ymlinstalls Mesa's software Vulkan on ubuntu-24.04 and runscargo ci-gpu: the wgpu layer, the presenter and the VPro display tests. The tests assert frame counts, not pictures.Worth a close look:
Presenter::present's two targets, and the wait on the previous submission that keeps at most two offscreen frames in flight.main.rs.wgpu.ymlis the right scope.Evidence, on x86-64 with an RTX 4090 and Vulkan, from a release build with a V12 Tezro at the IRIX greeter in ci mode with the offscreen renderer:
cargo ci-gpuselects pass on the RTX 4090. They also pass on lavapipe in Debian sid and ubuntu:24.04 (Mesa 25.2.8) containers, four at a time.cargo check --tests --binspasses with default features, jitv2, wgpu, and jitv2 + rex-gpu + rex-jit. Theodyssey,wgpuanduimodule tests pass, default and with wgpu.The windowed GPU path costs far more at idle than the CPU compose path: about 257 M instructions a minute against about 5 M. That is about 390 k per idle-refresh present. This branch does not address it.
Not verified:
VPRO_GPU=1, two VPro golden scenes differ from their goldens by one colour step, as they do on main. The CI profile leaves them out.