Repository navigation
o2: finish queued MTE transfers before a status read answers - #689
Open
jamesbraid wants to merge 2 commits into
Open
jamesbraid wants to merge 2 commits into
jamesbraid wants to merge 2 commits into
Conversation
On a busy host the O2 now and then stopped at the PROM's serial loader prompt, SL-9600-8E>, with nothing before it, where a good boot prints "Running power-on diagnostics" and reaches its menu. It happened once in the boot test on a macOS arm64 CI runner, never on an idle host. PROM 4.18 clears 32 KB at the base of RAM with the MTE (0xbfc061f0), reads the rendering engine's status once and goes on whatever it says: its wait at 0xbfc06400 tests a value that can never be negative. It then stores sp at 0xa0001008 as a cookie, copies its POST code to 0xa0004000, runs it, and drops into the serial loader if the cookie has changed. The engine runs on the CRIME-RE host thread, and a status read waited for it at most 100 us. When the host kept that thread off a processor for longer, the clear ran after the cookie store and zeroed it. Later still, it zeroed the POST code and the CPU hung at 0xbfc003a0 on a reserved instruction, printing nothing. With a sleep injected before each transfer (not committed), on a Linux host, release build with lightning and jitv2, phase A only: up to 1300 us the menu came up; 1400 and 1500 us gave SL-9600-8E> byte for byte as in CI, with physical 0x1008 reading 0; 1600 us and more hung in POST. With no injected delay, the CRIME-RE thread at SCHED_IDLE on a core running a busy loop hung 3 boots of 7. Stopping the whole process never reproduced it, as that stalls the CPU along with the engine. The engine now counts transfers queued and carried out, and a status read answers BUS_BUSY until they match, so the CPU retries it and cannot get past a transfer it started. After the change the same delays from 1300 us to 20 ms all reach the menu, as did 3 of 3 starved boots. The test queues a clear, starts the engine 200 ms later and fails before the change: the read answered with the memory still unwritten. The bounded wait now holds for primitives only, so a driver that only checks for room while drawing still overlaps the engine. MTE transfers do not. IRIX's MTE copy path reads the status for room right after starting a transfer, so a copy longer than 100 us now holds the CPU until it is done. Before, the read gave up after 100 us and reported a busy level. A synthetic back-to-back 8 KB clear plus status read took about 10.0 us per pair, against 8.6 us without the guard (release build, 20,000 pairs, best of 7, three runs each). That is an upper bound for the micro-pattern, not an X or GL cost. No X or GL benchmark exists in the tree to measure the real cost.
When the O2 PROM stopped at its serial loader in CI, boot-test left an empty phase-a.log and serial.log held only SL-9600-8E>. Nothing said where the PROM was parked, so the failure could not be told from a POST hang or a missed menu, and it took a local reproduction to find the cause. phase-a.log was empty because a wait that times out takes none of the output. On a phase A failure the test now drains what the PROM printed into phase-a.log, stops the CPU through the monitor and writes its registers to phase-a-cpu.log. On the O2 it also dumps the words at physical 0x1000 and 0x4000, where the PROM keeps its stack cookie and POST code: zeros there point at a memory clear that landed late, intact words at some other route into the serial loader. Both files match the artifact upload's *.log pattern. Checked with a PROM whose firmware segment fails its checksum: the serial loader prompt landed in phase-a.log and the PC, 0xbfc013ac, in phase-a-cpu.log. The same PROM parked at the prompt shows the cookie intact (0xffffffff80000fa8 at 0x1008) and POST's first instruction at 0x4000, so the dump tells that route from a late clear.
jamesbraid
force-pushed
the
fix/o2-mte-status-wait
branch
from
November 13, 2026 04:00
75d852e to
93c8252
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
On a busy host the O2 sometimes stopped at the PROM's serial loader prompt,
SL-9600-8E>, before its menu. It happened once in the boot test on a macOS arm64 runner. A later stage of the same bug hangs the CPU with nothing on serial.The cause is a race between the PROM and the rendering engine's host thread. PROM 4.18 clears the first 32 KB of RAM with the MTE and reads the engine's status once. Its wait loop can never loop, so it carries on regardless of what the read returned. Next it stores a stack cookie in that memory and copies its POST code there. iris runs the engine on the CRIME-RE thread, and a status read waited at most 100 us for it. If the host kept that thread off a processor for longer, the clear ran late. It zeroed the cookie, which sends the PROM to the serial loader, or zeroed the POST code, which hangs the CPU.
Changes
BUS_BUSYuntil the two counts match, and the CPU retries the read. The bounded 100 us / 20 ms wait now holds for primitives only, so a driver that only checks for room while drawing still overlaps with the engine.iris-ci boot-testnow keeps everything the PROM printed inphase-a.log. It also stops the CPU through the monitor and writes the registers tophase-a-cpu.log. On the O2 it adds the words at physical 0x1000 and 0x4000, the PROM's stack cookie and POST code. Zeros there mean a late clear. Intact words mean some other route into the serial loader. The existing*.logartifact pattern picks up both files.Worth a look
phase-a-cpu.logsettle it on the next occurrence.Verification
Unit test
a_status_read_answers_only_once_a_queued_transfer_is_donequeues a clear and starts the engine 200 ms later. It fails before the fix and passes after. The O2 and iris-ci unit tests pass with and without the CI features.I put a sleep before each transfer (an experiment only, not committed) and ran phase A on Linux:
SL-9600-8E>, same bytes as CI, cookie reads 0With no injected sleep, I set the CRIME-RE thread to
SCHED_IDLEon a core running a busy loop. Before the fix, 3 of 7 boots hung. After it, phase A reached the menu 3 of 3 times.Full O2 6.5.30 boot test, Linux, CI features: 3 of 3 passed after the fix, with serial login at 65 s and a total of 148–151 s each time. Before the fix: 150 s in one run. The other run took 444 s, with a 360 s serial login I didn't investigate.
The full boot test also passed after the fix with every run's CRIME-RE thread starved (888 s) and with a 1.4 ms sleep before every transfer (176 s).
cargo check --workspace --all-targetsis clean, default and with jitv2.The whole library suite has 7 to 10 failures in the build container, in file-backed tests (CHD, NVRAM and snapshot files). It fails the same way on main and the failing tests pass when run alone.
Not run: macOS and Windows hosts, IRIX graphics workloads beyond the login screen, NetBSD and Linux O2 guests.