Repository navigation
Inspect: off-screen elements, wait snapshots past the budget, expect from the snapshot, web accessibility restore - #238
Merged
Merged
Conversation
…pect from the snapshot, restore web accessibility - #224: expect on a snapshot-targeted click, key, drag or insert takes its pid, and window for window, element and text conditions, from the snapshot; expect's own refusals say they came from expect. - #178: an unscoped inspection skips elements wholly outside the window, or outside the scroll or web area clipping them, with their subtrees, and counts them in offscreen_omitted. A scope still reads everything. - #232: when element_appears matches an element the unscoped inspection doesn't list, the wait inspects again with the scoped budget, listing the outermost ancestor subtree of the match that fits 200 elements. - #231: web accessibility the host turned on is turned back off after 10 minutes without a read of that app, or at once with web_accessibility false (reported as restored). One found on is never turned off. Why: native2/Sources/DesktopMac/Elements.swift | visibleOnly is for unscoped inspection only; scoped reads, desktop_read and wait matching must keep off-screen content, which an agent reaches by scope or scrolling Why: native2/Sources/DesktopHost/Scope.swift | the idle restore waits as long as a snapshot lives (600 s), so no live snapshot holds web elements Chrome has torn down Why: native2/Sources/DesktopMac/WebAccessibility.swift | only launches this host turned on are turned off; one found on may belong to VoiceOver or another assistive app Why: native2/Sources/DesktopHost/Tools.swift | expect is filled from the snapshot before it is validated, and a dialog condition gets only the pid so it still sees dialogs in new windows
Contributor
Author
|
Live status: not yet verified live. The dev app built from this branch, but the macOS session was locked for the whole test window (more than 20 minutes), so every Accessibility read returned session_locked or an incomplete read. The checks are scripted in /tmp/dt-live/live.zsh on Nate's Mac. It takes the dev-app lock with zsystem flock and opens the fixtures in native2/fixture/inspect in temp-profile Chrome instances. Run it with |
Contributor
Author
|
Checked live on a dev app built from all three fix branches merged together:
Chrome doesn't expose off-screen content on |
Merged
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #224
Fixes #232
Fixes #178
Fixes #231
#224:
expecton a snapshot-targeted actionCause:
expectwas validated as a standalonedesktop_waitcondition, sopid(andwindow) were required even though the action's target came from its snapshot.Fix:
Tools.wholefills the condition's missingpid, andwindowforwindow_goneand element and text conditions, from the action's snapshot before validating it (Tools.expectTarget). Dialog conditions get only the pid, so they still read the whole app and see a dialog that opens in a new window. A condition that names another pid keeps it and doesn't borrow this window. Refusals fromexpectnow start withexpect:, sodesktop_system_key(which has no snapshot) saysexpect: pid must be a positive integer.#178: off-screen elements spend the budget
Cause: the AX walk emitted every element in pre-order up to 200, wherever it was.
Fix: an unscoped inspection's walk (
ElementBudget.visibleOnly) skips any element that lies wholly outside its clip, along with its subtree, and counts them inoffscreen_omitted. The clip starts as the window and narrows at eachAXScrollAreaandAXWebArea. An element on the clip's edge is kept, and so is a focused element. Element order, type grouping and truncation are unchanged, and the screenshot is still taken after the walk (#159).What stays reachable:
scopeinspections,desktop_readand the wait's matching read keep everything. A scroll or web area that intersects the viewport is still listed, so an agent can scope into it and get its off-screen content whole, or scroll and inspect again.#232: wait matches past the budget but returns no snapshot
Cause:
confirm()took an unscoped 200-element inspection and reportedsnapshot_id: nullwhen the match wasn't in it.Fix: when the unscoped inspection misses the match, the wait inspects again with the scoped budget. It lists the subtree of the match's outermost ancestor that holds at most 200 elements (
listingRoot), and the result names that root ininspection.scope. The snapshot stores the whole wide walk, so the element IDs are actionable.#231: Chrome web accessibility stays on
Cause: the host set
AXManualAccessibilityorAXEnhancedUserInterfaceonce per app launch and never set it back.Fix: the host records which launches it turned on, and with which attribute (
WebAccessibilityLedger).web_accessibility: falseon inspect, read or wait now also turns it off at once, if this host turned it on, and reportsrestored.Decision for the lead (trade-offs)
false= turn off (chosen). This needs no schema change and is discoverable. The cost is semantic:falseused to mean "leave as is". With two agents on one Chrome, one agent passingfalsemakes the other's next read re-enable it.web_accessibility: "restore", or a dedicated cleanup tool. That's clearer, but it's a schema change in shared files.false. Revisit disconnect-based restore if agents report slowness during long sessions.Tests
swift test --package-path native2passes, with newInspectScopeTests.swift(ExpectTarget, VisibleClip, ListingRoot and WebAccessibilityLedger suites).npm run check,npm testandnode scripts/contract.mjs --checkpass.native2/fixture/inspect/.