Skip to content

Inspect: off-screen elements, wait snapshots past the budget, expect from the snapshot, web accessibility restore - #238

Merged
iamnbutler merged 2 commits into
mainfrom
fix-inspect
Nov 3, 2026
Merged

iamnbutler merged 2 commits into
mainfrom
fix-inspect

Conversation

@iamnbutler

Copy link
Copy Markdown
Contributor

Fixes #224
Fixes #232
Fixes #178
Fixes #231

#224: expect on a snapshot-targeted action

Cause: expect was validated as a standalone desktop_wait condition, so pid (and window) were required even though the action's target came from its snapshot.
Fix: Tools.whole fills the condition's missing pid, and window for window_gone and element and text conditions, from the action's snapshot before validating it (Tools.expectTarget). Dialog conditions get only the pid, so they still read the whole app and see a dialog that opens in a new window. A condition that names another pid keeps it and doesn't borrow this window. Refusals from expect now start with expect: , so desktop_system_key (which has no snapshot) says expect: pid must be a positive integer.

#178: off-screen elements spend the budget

Cause: the AX walk emitted every element in pre-order up to 200, wherever it was.
Fix: an unscoped inspection's walk (ElementBudget.visibleOnly) skips any element that lies wholly outside its clip, along with its subtree, and counts them in offscreen_omitted. The clip starts as the window and narrows at each AXScrollArea and AXWebArea. An element on the clip's edge is kept, and so is a focused element. Element order, type grouping and truncation are unchanged, and the screenshot is still taken after the walk (#159).
What stays reachable: scope inspections, desktop_read and the wait's matching read keep everything. A scroll or web area that intersects the viewport is still listed, so an agent can scope into it and get its off-screen content whole, or scroll and inspect again.

#232: wait matches past the budget but returns no snapshot

Cause: confirm() took an unscoped 200-element inspection and reported snapshot_id: null when the match wasn't in it.
Fix: when the unscoped inspection misses the match, the wait inspects again with the scoped budget. It lists the subtree of the match's outermost ancestor that holds at most 200 elements (listingRoot), and the result names that root in inspection.scope. The snapshot stores the whole wide walk, so the element IDs are actionable.

#231: Chrome web accessibility stays on

Cause: the host set AXManualAccessibility or AXEnhancedUserInterface once per app launch and never set it back.
Fix: the host records which launches it turned on, and with which attribute (WebAccessibilityLedger).

  • Idle restore: after 10 minutes with no read of that app that relied on it, the host sets that attribute back to false. Each inspect, wait poll and post-action observation counts as a read. 10 minutes is the snapshot lifetime, so no live snapshot holds web elements that Chrome has torn down.
  • Explicit: web_accessibility: false on inspect, read or wait now also turns it off at once, if this host turned it on, and reports restored.
  • Found on: web accessibility that was already on (VoiceOver, another AT) is never turned off.

Decision for the lead (trade-offs)

  • Idle timeout (chosen, 10 min). It needs no agent cooperation, and it covers agents that never clean up. It doesn't help while the agent works, and a heavy page stays slow for up to about 11 minutes after the last read. A shorter timeout would leave live snapshots pointing at dropped web elements, and each re-enable costs 0.5–3 s plus Chrome rebuilding its tree.
  • false = turn off (chosen). This needs no schema change and is discoverable. The cost is semantic: false used to mean "leave as is". With two agents on one Chrome, one agent passing false makes the other's next read re-enable it.
    • Alternative: a separate value such as web_accessibility: "restore", or a dedicated cleanup tool. That's clearer, but it's a schema change in shared files.
  • Not done: restoring on agent disconnect, since the host can't tell when "work is done" for an MCP session. Restoring when the host quits isn't done either.
  • Recommendation: keep idle plus false. Revisit disconnect-based restore if agents report slowness during long sessions.

Tests

  • swift test --package-path native2 passes, with new InspectScopeTests.swift (ExpectTarget, VisibleClip, ListingRoot and WebAccessibilityLedger suites).
  • npm run check, npm test and node scripts/contract.mjs --check pass.
  • Live: see the comment below. Fixture pages are in native2/fixture/inspect/.

…pect from the snapshot, restore web accessibility

- #224: expect on a snapshot-targeted click, key, drag or insert takes its pid, and window for window, element
  and text conditions, from the snapshot; expect's own refusals say they came from expect.
- #178: an unscoped inspection skips elements wholly outside the window, or outside the scroll or web area
  clipping them, with their subtrees, and counts them in offscreen_omitted. A scope still reads everything.
- #232: when element_appears matches an element the unscoped inspection doesn't list, the wait inspects again
  with the scoped budget, listing the outermost ancestor subtree of the match that fits 200 elements.
- #231: web accessibility the host turned on is turned back off after 10 minutes without a read of that app,
  or at once with web_accessibility false (reported as restored). One found on is never turned off.

Why: native2/Sources/DesktopMac/Elements.swift | visibleOnly is for unscoped inspection only; scoped reads, desktop_read and wait matching must keep off-screen content, which an agent reaches by scope or scrolling
Why: native2/Sources/DesktopHost/Scope.swift | the idle restore waits as long as a snapshot lives (600 s), so no live snapshot holds web elements Chrome has torn down
Why: native2/Sources/DesktopMac/WebAccessibility.swift | only launches this host turned on are turned off; one found on may belong to VoiceOver or another assistive app
Why: native2/Sources/DesktopHost/Tools.swift | expect is filled from the snapshot before it is validated, and a dialog condition gets only the pid so it still sees dialogs in new windows
@iamnbutler

Copy link
Copy Markdown
Contributor Author

Live status: not yet verified live. The dev app built from this branch, but the macOS session was locked for the whole test window (more than 20 minutes), so every Accessibility read returned session_locked or an incomplete read. The checks are scripted in /tmp/dt-live/live.zsh on Nate's Mac. It takes the dev-app lock with zsystem flock and opens the fixtures in native2/fixture/inspect in temp-profile Chrome instances. Run it with zsh /tmp/dt-live/live.zsh once the session is unlocked.

@iamnbutler
iamnbutler merged commit c9be4ae into main Nov 3, 2026
2 checks passed
@iamnbutler

Copy link
Copy Markdown
Contributor Author

Checked live on a dev app built from all three fix branches merged together:

Chrome doesn't expose off-screen content on long.html at all (0.19.0 also reads only 84 elements there), so that fixture can't test #178.

@iamnbutler iamnbutler mentioned this pull request Nov 3, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment