Repository navigation
feat: let the user choose a screen or window in the macOS system picker - #592
Merged
Merged
Conversation
VideoDesktopSource.setUseSystemPicker(true) makes start() show the operating system's own picker, instead of capturing the source set with setSourceId(). On macOS 14 and newer that is ScreenCaptureKit's SCContentSharingPicker, offering a screen or one or more windows; under Wayland it is the desktop portal, as before. Both come through DesktopCapturer::CreateGenericCapturer. ScreenCaptureKit draws the cursor, which DesktopAndCursorComposer would misplace on a picked window. Where isSystemPickerAvailable() is false, the source ends instead of capturing something the user did not choose. The default is unchanged. A source now ends when its capturer gives up for good: the user cancelled the picker or stopped sharing, or the captured window was closed. Before, capture stopped but the track stayed live. The ended listeners run on the capture thread, as for a missing backend, so that thread now keeps the source alive until it ends, and stop() called on it does not join it (#186). Should the last reference go there, a short-lived thread joins it. dispose() ends the capture without waiting, even while a track still holds the source. DesktopPickerExample shows the picker, and the desktop capture guide describes it.
DesktopCapturer.setUseSystemPicker(true), set before start(), makes start() show the operating system's own picker instead of capturing the source chosen with selectSource(): ScreenCaptureKit's SCContentSharingPicker on macOS 14 and newer, the desktop portal under Wayland, as VideoDesktopSource already does. Until the user has chosen, captureFrame() reports ERROR_TEMPORARY; a cancelled picker or stopped sharing reports ERROR_PERMANENT, as does every frame where isSystemPickerAvailable() is false. Setting it after start() throws IllegalStateException. Changing the setting replaces the native capturer, keeping the maximum frame rate. VideoDesktopSource now asks DesktopCapturer whether a picker is available, so both answer alike.
DesktopCapturerPickerExample starts a ScreenCapturer set to use the system picker, calls captureFrame() at the frame rate on the one thread that uses the capturer, and shows the frames in a VideoView. Its status line tells a pending choice from a cancelled picker or stopped sharing.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Follows up on #344, which suggested
allow_sck_system_pickerso that macOS's own picker chooses the screen or window, as the Wayland portal does.Builds on the ScreenCaptureKit window and screen capture of #591.
The system picker, opt-in
SCContentSharingPicker, offering a screen or one or more windows. WebRTC fixes these modes,SingleDisplay | MultipleWindows.VideoTrackDesktopSource::capture()setsallow_sck_system_pickerand takes the capturer fromDesktopCapturer::CreateGenericCapturer, as the Wayland path already does. ScreenCaptureKit draws the cursor (prefer_cursor_embedded), sinceDesktopAndCursorComposerwould misplace it on a picked window.isSystemPickerAvailable()is false, and a source set to use the picker ends right away. The fallback is not the source ID: its default, -1, would share the whole desktop.It is opt-in, and the default is unchanged. Apps with their own source list would otherwise see a second chooser and have
setSourceIdignored, and the picker needs a person to answer it.Limits from WebRTC: show the picker for one source at a time, since every source waiting on it receives the user's choice. The app cannot reopen the picker on a running source;
stop()andstart()show a new one.The picker on
ScreenCapturerandWindowCapturerThe same option is available on
DesktopCapturer, for apps that take frames themselves and don't use a track:captureFrame()reportsERROR_TEMPORARY. A cancelled picker or stopped sharing reportsERROR_PERMANENT. WhereisSystemPickerAvailable()is false, every frame reportsERROR_PERMANENTinstead of capturing something the user did not choose.setUseSystemPicker()afterstart()throwsIllegalStateException, as a secondstart()does.DesktopCapturer::createCapturer(). The maximum frame rate is kept, but a selected source is forgotten.ScreenCapturerand aWindowCapturer, since it offers both screens and windows. A picked window's frames have no window border to crop, soisWindowCapturer()is false with the picker.VideoTrackDesktopSource::isSystemPickerAvailable()now asksjni::DesktopCapturer, so both APIs give the same answer.selectSource()path works as before.DesktopCapturerPickerExampleis a small Swing app that shows the picked content live, and the screen and window capturer docs describe the option.The end of capture
Cancelling the picker, stopping sharing from the menu bar (macOS 14.4+), and a picker that fails to open all reach
VideoTrackDesktopSourceasERROR_PERMANENT. So far that only stopped the capture loop, and the track stayedLIVE. Now:stop()called on that thread no longer joins the thread from itself, which would hitRTC_CHECKon POSIX (Native Crash related to Window/Video Capture #186). The nextstop(), or the destructor, joins it.dispose()ends the capture without waiting for the thread, consistent with the non-blocking dispose inJNI_VideoDesktopSource. Before, a track holding a disposed source kept receiving frames.Testing
macOS 14.5, Apple silicon:
mvn -pl webrtc test: 1455 tests, 0 failures, 49 skipped (platform and hardware assumptions, plusendsWithoutPickeron a Mac with a picker).-Pjni-checkonVideoDesktopSourceTestsandDesktopCapturerTests: noFATAL ERROR.New tests:
systemPickerAvailability: true on macOS 14+ and on Wayland, false otherwise.endsWithoutPicker: the source ends at once where no picker exists. It runs on Windows and X11 and skips on macOS 14+.closedWindowEndsTrack(wasclosedWindowStopsCapture): the track ends, and the frames stop.endedListenerMayStopAndDispose: the ended listener callsstop()anddispose()on the capture thread, without a crash.disposeEndsCapture: no frames reach a track once its source is disposed.DesktopCapturerTests.systemPickerAvailability: matchesVideoDesktopSource.isSystemPickerAvailable().pickerAfterStartIsRejected:setUseSystemPicker()afterstart()throws.noPickerReportsPermanentError: where no picker exists, the first frame reportsERROR_PERMANENT. It skips on macOS 14+.pickerCanBeTurnedOff: turning the picker on and off again gives a capturer that lists the same screens.After the
DesktopCapturerchange,DesktopCapturerTestsandVideoDesktopSourceTestsran again: 22 tests, 0 failures, 4 skipped (systems without a picker or without a backend).Verified by hand: the picker appears. It was presented from the capture thread in a plain JVM without AWT, and a screenshot showed macOS's "share this screen" overlay and the menu bar sharing indicator.
stop()while the picker is open destroys the capturer cleanly, and the track staysLIVE.Verified by hand with
DesktopCapturerPickerExample(Swing,ScreenCapturer,start()andcaptureFrame()on one executor thread): choosing a screen or windows delivers frames, and cancelling the picker and stopping sharing reportERROR_PERMANENT.Still to check by hand, with
DesktopPickerExample. This is why the PR is a draft. The picker needs a click, so it stays out of the automated tests.-XstartOnFirstThread, where the main thread runs no event loop: the picker appears, a screen (3840x2160) and a window (1920x997) deliver frames at 15 fps, and cancelling the picker or stopping sharing ends the track. Changing the choice from the menu bar was not repeated with this flag.Not tested: macOS 15 and newer, and whether the picker spares an app the Screen Recording permission.