Skip to content

Pointer: foreground plain click, context menu items in one call, Chromium routed click - #239

Merged
iamnbutler merged 9 commits into
mainfrom
fix-pointer
Nov 3, 2026
Merged

iamnbutler merged 9 commits into
mainfrom
fix-pointer

Conversation

@iamnbutler

Copy link
Copy Markdown
Contributor

Fixes #225
Fixes #227
Fixes #230

#225: background click isn't a mouse event

Cause. A background single click is an Accessibility press, or, on a list row, an AXSelected write. The app gets no mouse event, so a selection handler that checks NSApp.currentEvent (the listfix fixture logs event=other(13)) ignores it, and the result still said completed.
Fix. desktop_click takes foreground: true for a plain single, double or triple click with the real pointer, through the same path as modifier clicks: activate, click, restore pointer and focus, result unknown. The description says a background single click is an Accessibility press or selection that such apps may ignore, and a background row selection carries a note saying so. Foreground right and middle clicks are refused, since those are already pointer events.

#227: context menu closes before an item can be pressed

Cause. An app in the background closes its context menu as soon as other input arrives, often before the next call.
Fix. A right click now finds the open AXMenu (the app's children, else the AXMenu in its pop-up-menu-level window) and lists it in context_menu.items. menu_path (exact titles, as desktop_menu takes them) presses that item in the same call. A wrong, ambiguous, disabled or submenu title closes the menu with AXCancel, chooses nothing, and returns unknown (the right click went out) with the titles found. A Paste item is refused while a secure field has focus, as desktop_menu does. An AXShowMenu still blocked on its open menu now counts as delivered, so a right click that opened a menu is completed, no longer unknown.

#230: Chrome clicks refused with "No pressable accessibility element"

Cause. Chromium exposes AXPress only on elements with their own click listener. Pages that listen for clicks once at the root, such as Google's jsaction apps and many frameworks, expose nothing pressable on their rows, links and ⋮ buttons. Reproduced with a local page that delegates clicks: a row by element ID, a row's ⋮ and an email "link" by point were all refused.
Fix. In Chromium apps (routesThroughSkyLight), when no candidate at the point is pressable, the click is sent as a single pointer click routed to the window (SLEventPostToPid, the route double and right clicks already use). Chromium handles it in the background with isTrusted true. The result's note says so. Pressable elements are still pressed through Accessibility first. Other apps keep the refusal, which now suggests foreground: true.

Tests

  • Swift: ContextMenuTests (path resolution, item listing, argument refusals). swift test --package-path native2 passes. npm run check, npm test and node scripts/contract.mjs --check pass.
  • Live: see the comments below.

@iamnbutler

Copy link
Copy Markdown
Contributor Author

Live run on this Mac, dev app built from 54f63c3. Chrome 156 ran with a temp profile; native2/fixture/listfix.swift is the SwiftUI list fixture. Both apps were in the background.

  • Background click returns completed, but apps that check NSApp.currentEvent ignore it #225: a background click on the Bravo row returned completed with the selection note. The fixture logged selection ["Bravo"] event=other(13), so the app ignored it, as the note says. kind: right with foreground: true was refused before input. foreground: true on a row ended unknown: "listfix didn't become frontmost". An existing modifiers: ["command"] click on the same fixture failed the same way, so this is activation on this Mac or with this unbundled fixture, not the new path. The session locked soon after, so I couldn't recheck it.
  • Context menu from a background right-click closes before its items can be pressed #227: a right click on Alpha listed context_menu.items (Rename Alpha, Delete Alpha, More [submenu], More > Archive Alpha), and the menu was still open a second later. menu_path: ["Rename Alpha"] returned completed and the fixture logged acted rename Alpha. A point right click with ["More", "Archive Charlie"] returned completed and logged acted archive Charlie. ["Nope"] returned unknown with the titles found, the menu was closed, and nothing was chosen. The plain right click came back unknown instead of completed, because AXShowMenu was still blocked on the open menu. That is fixed in Click.swift (a menu seen open counts as delivered), but the fix hasn't been rechecked live.
  • Background click refused on clickable Chrome content: Cloud console links, Calendar settings rows by element ID #230: on a local page whose clicks are handled at the document root (like jsaction), the old build refused a row by element ID, the ⋮ by point and the email span by point. The new build returned completed with the routed-click note for all four targets: Calendar-style row by ID, service-account row by ID, ⋮ and email by point. The page logged email1 trusted=true and row1 trusted=true. Chrome's Accessibility text for the log lagged far behind the page, so the observation right after each click didn't show it, and the cal1 and kebab1 single clicks can't be told apart from earlier double clicks in that log. The page now writes each click into its title for a cleaner recheck.

swift test, npm run check, npm test and contract.mjs --check pass, and CI passes (javascript, native).

…mium routed click

- desktop_click foreground true sends a plain single, double or triple click with the real pointer (#225); a background row selection says in its note that it isn't a mouse event.
- A right click lists the context menu it opened, and menu_path presses one of its items in the same call, since a background app closes the menu when other input arrives (#227).
- In Chromium apps, a point with nothing pressable gets a pointer click routed to the window instead of a refusal (#230).

Why: native2/Sources/DesktopMac/Pointer.swift | Chromium exposes AXPress only on elements with their own click listener; pages that delegate clicks at the root (jsaction, frameworks) expose nothing, so a Chromium click with no pressable candidate must fall back to a routed pointer click, which Chromium accepts in the background
Why: native2/Sources/DesktopHost/Click.swift | a background app's context menu closes when other input arrives, so items must be pressed in the call that opened it; AXShowMenu blocks while the menu is open, so a menu seen open is a finished delivery
…en that can't be confirmed (#230)

A Chrome launched in the background and never activated drops routed left clicks: AppKit discards a mouseDown in a window that isn't key in its app, while Chrome's Accessibility focus names the window either way. The fallback now makes the window key with prepareBackgroundWindow first, refuses when that is refused (nothing sent), and is unknown unless the app then takes itself for active with the window focused.

Why: native2/Sources/DesktopMac/Pointer.swift | a routed left click into a Chromium window that isn't key in its app is dropped (right clicks still arrive); AXFocusedWindow can't tell, only the app's own AXFrontmost after prepareBackgroundWindow confirms it
@iamnbutler
iamnbutler merged commit 4d20a02 into main Nov 3, 2026
2 checks passed
@iamnbutler iamnbutler mentioned this pull request Nov 3, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

1 participant