Repository navigation
feat: capture macOS windows with ScreenCaptureKit, and close the screen capture gaps - #591
Merged
Merged
Conversation
Windows were captured with WebRTC's WindowCapturerMac, which takes a CGWindowListCreateImage snapshot for every frame. Apple deprecated that call in macOS 14 and made it unavailable to apps built for macOS 15, and WebRTC, also at branch-heads/8059, has no ScreenCaptureKit window capturer outside its system picker. On macOS 14 and newer, VideoDesktopSource and WindowCapturer now capture a window, chosen by the same CGWindowID as before, with a ScreenCaptureKit stream of our own. Frames arrive as IOSurfaces cropped to the window. The stream follows the window's size, ScreenCaptureKit draws the cursor, and a minimized window keeps its last image. For a 2560x1600 pixel window at 30 fps, the capturing process used about 23% of a core instead of 35% (Apple silicon, macOS 14.5). Windows of the capturing process itself, and windows ScreenCaptureKit cannot capture, for example without screen recording permission, still go to WindowCapturerMac. A stream kept a closed window of its own process alive, so that capture never ended, and Core Graphics needs no permission for such windows. For other processes, ScreenCaptureKit stops the stream when the window closes, and capture stops as before. Window lists and source IDs are unchanged. Issue #344 reports window sharing failing on macOS; its cause there is not known yet.
…creenCaptureKit newer, by setting allow_sck_capturer for VideoDesktopSource. That left three gaps, all closed by a screen capturer that picks its backend when started (MacOSScreenCapturer): - The whole desktop (source ID -1, all screens in one image): ScreenCapturerSck cannot capture it, and silently delivered only the first screen. The CGDisplayStream capturer captures it again. - ScreenCapturer.start()/captureFrame() still captured with the CGDisplayStream capturer, which can freeze on its first frame (#344), because ScreenCaptureKit lists no screens without its system picker. Screens are now listed by the CGDisplayStream capturer, as before, and a single screen is captured with ScreenCaptureKit, as in VideoDesktopSource. Both name a screen by its CGDirectDisplayID. - ScreenCapturerSck's frames do not say where the screen lies on the desktop, while DesktopAndCursorComposer places its cursor in desktop coordinates, so on any screen but the first the cursor was missing or misplaced. ScreenCaptureKit now draws the cursor itself. Before macOS 14, every screen is still captured with the CGDisplayStream capturer.
ownWindowFramesFollowWindow failed once in a full run: after the window turned white, its frames stayed black. Its capture was running, but the window had not been redrawn. A window of a test process that is not in front may not be redrawn in time, and it cannot be put on top, since WindowCapturer lists only windows of the normal level. The test now checks that the window is captured at its size, and that the capture ends when the window closes. Content is checked on another process's window, which is now ordered to the front.
7 tasks done
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Follows up on #344, where sharing a single window didn't work at all for a macOS user. That cause isn't known yet: the reporter hasn't given their macOS version or permission state. This PR moves window capture off the API that #354 had to leave in place.
Windows are captured with ScreenCaptureKit on macOS 14 and newer
On macOS,
VideoDesktopSourceandWindowCapturercaptured windows with WebRTC'sWindowCapturerMac, which takes aCGWindowListCreateImagesnapshot for every frame. Apple deprecated that call in macOS 14, and it is unavailable to apps built for macOS 15 and newer. WebRTC at branch-heads/8059 still has no ScreenCaptureKit window capturer outside its system picker, so this adds one of our own:ScreenCaptureKitWindowCapturer, awebrtc::DesktopCapturerinsrc/media/video/desktop/macos/.It looks up the selected
CGWindowIDinSCShareableContentand streams that window withinitWithDesktopIndependentWindow:. Frames arrive as IOSurfaces and are wrapped and cropped to the window with WebRTC'sDesktopFrameIOSurface.scalesToFitoff, a window that outgrew the output is scaled down until the nextCaptureFrame()reconfigures the output to the window's on-screen size (SCStreamFrameInfoScreenRect). One reconfiguration per resize, e.g. 800x600 → 1600x1000 → 600x500 pixels.DesktopAndCursorComposerdoesn't draw a second one. The composer works in screen coordinates and would misplace the cursor on a window's frame.ignoreShadowsSingleWindow), likekCGWindowImageBoundsIgnoreFramingbefore.CaptureFrame()reportsERROR_PERMANENT, and capture stops as before.VideoDesktopSourceandjni::DesktopCaptureruse it for windows on macOS 14 and newer. Window lists and source IDs are unchanged:GetSourceList()is WebRTC'sGetWindowList(), as inWindowCapturerMac.What still goes to
WindowCapturerMac