Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
23 changes: 23 additions & 0 deletions docs/guide/examples.md
Original file line number Diff line number Diff line change
Expand Up @@ -148,6 +148,29 @@ Each audio sender gets the audio of its own track, so the microphone keeps its e
mvn exec:java -D"exec.mainClass=dev.onvoid.webrtc.examples.ScreenShareAudioExample"
```

## Desktop Picker

The [`DesktopPickerExample`](https://github.com/devopvoid/webrtc-java/blob/master/webrtc-examples/src/main/java/dev/onvoid/webrtc/examples/DesktopPickerExample.java) lets the user choose a screen or window in the operating system's own picker, on macOS 14 and newer and under Wayland.

**Key features demonstrated:**
- Checking whether the system offers a picker
- Showing the picker by starting a `VideoDesktopSource`
- Reacting to the end of the track when the picker is cancelled or sharing stops

## Desktop Capturer Picker

The [`DesktopCapturerPickerExample`](https://github.com/devopvoid/webrtc-java/blob/master/webrtc-examples/src/main/java/dev/onvoid/webrtc/examples/DesktopCapturerPickerExample.java) is a small Swing application that takes frames from a `ScreenCapturer` without a video track, after the user has chosen a screen or window in the operating system's picker.

**Key features demonstrated:**
- Setting `setUseSystemPicker(true)` before `start()`, which shows the picker
- Calling `captureFrame()` at the frame rate, on the one thread that uses the capturer
- Showing the frames in a Swing `VideoView`
- Telling a pending choice (`ERROR_TEMPORARY`) from a cancelled picker or stopped sharing (`ERROR_PERMANENT`)

```bash
mvn exec:java -D"exec.mainClass=dev.onvoid.webrtc.examples.DesktopCapturerPickerExample"
```

## Media File

The [`MediaFileExample`](https://github.com/devopvoid/webrtc-java/blob/master/webrtc-examples/src/main/java/dev/onvoid/webrtc/examples/MediaFileExample.java) demonstrates how to send a media file over a peer connection, in place of a camera and a microphone. See the [Media Files](/guide/media/media-files) guide for the API it uses.
Expand Down
33 changes: 29 additions & 4 deletions docs/guide/video/desktop-capture.md
Original file line number Diff line number Diff line change
Expand Up @@ -70,6 +70,27 @@ videoSource.setSourceId(windowId, true);
videoSource.start();
```

### Letting the User Choose in the System Picker

Instead of listing sources yourself, you can let the user choose a screen or window in the operating system's own picker. It is available on macOS 14 and newer, where it is ScreenCaptureKit's content sharing picker, and on Linux under Wayland, where the desktop portal's picker is always used:

```java
VideoDesktopSource videoSource = new VideoDesktopSource();
videoSource.setFrameRate(30);

if (VideoDesktopSource.isSystemPickerAvailable()) {
// The picker chooses the source; no setSourceId() is needed.
videoSource.setUseSystemPicker(true);
}
else {
videoSource.setSourceId(screenId, false);
}
```

`start()` shows the picker and returns at once. No frames arrive until the user has made a choice. The user can change it later from the macOS menu bar, and the frames follow. If the user cancels the picker, or stops sharing from the menu bar, the source ends (see [Handling the End of Capture](#handling-the-end-of-capture)). Where `isSystemPickerAvailable()` is false, a source set to use the picker ends right away instead of capturing.

On macOS the picker offers a whole screen or one or more windows. Show it for one source at a time: sources waiting on it at the same time all receive the user's choice. To take frames yourself rather than through a track, `ScreenCapturer` and `WindowCapturer` offer the same picker (see [Screen Capturer](/tools/desktop/screen-capturer#letting-the-user-choose-in-the-system-picker)). The [`DesktopPickerExample`](https://github.com/devopvoid/webrtc-java/blob/master/webrtc-examples/src/main/java/dev/onvoid/webrtc/examples/DesktopPickerExample.java) shows the picker and reports the frames it receives.

### Creating a Video Track with the Desktop Source

Once you have configured your desktop video source, you can create a video track:
Expand Down Expand Up @@ -153,16 +174,20 @@ videoSource.stop();
videoSource.dispose();
```

### Handling Capture Failures
### Handling the End of Capture

`start()` returns before capturing begins. If capturing cannot begin, because no capture backend is available or the selected source does not exist, the source ends and its tracks with it. Create the track and add the listener before calling `start()`, as a listener added later misses this:
`start()` returns before capturing begins. The source ends, and its tracks with it, when capturing cannot begin or cannot go on: no capture backend is available, the selected source does not exist, the captured window was closed, or the user cancelled the system picker or stopped sharing. Create the track and add the listener before calling `start()`, as a listener added later misses an end that comes at once:

```java
videoTrack.addTrackEndedListener(track -> {
// No frames will arrive from this source.
// No more frames will arrive from this source. The listener runs on the
// capture thread; stopping or disposing the source here is safe.
videoSource.stop();
});
```

`dispose()` ends the capture as well, even while a track still holds the source.

### Processing Frames

A `VideoDesktopSource` can pass its frames through a `VideoProcessor` before they reach its tracks, for example to blank out a part of the screen. It works as for a camera; see [Changing Frames Before They Are Sent](/guide/video/camera-capture#changing-frames-before-they-are-sent).
Expand All @@ -188,7 +213,7 @@ windowCapturer.dispose();

On macOS 14 and newer, screens and windows are captured with ScreenCaptureKit; older releases use the earlier Core Graphics capture. Both need the app to have Screen Recording permission (System Settings → Privacy & Security → Screen & System Audio Recording). Without it, the windows of other apps are listed without their titles and their content cannot be captured. Windows of the app's own process need no permission, and are captured with Core Graphics.

While a captured window is minimized or hidden, its last image repeats. When it is closed, capture stops.
While a captured window is minimized or hidden, its last image repeats. When it is closed, the source ends.

## Conclusion

Expand Down
32 changes: 32 additions & 0 deletions docs/tools/desktop/screen-capturer.md
Original file line number Diff line number Diff line change
Expand Up @@ -16,6 +16,8 @@ Key methods (inherited from `DesktopCapturer`):
- `void selectSource(DesktopSource source)` – choose the screen to capture
- `void setFocusSelectedSource(boolean focus)` – try to focus the selected source during capture
- `void setMaxFrameRate(int maxFrameRate)` – cap the capture FPS
- `void setUseSystemPicker(boolean usePicker)` – let the user choose a screen or windows in the operating system's picker, shown by `start`, instead of `selectSource`; set it before `start`, a later call throws `IllegalStateException`
- `static boolean isSystemPickerAvailable()` – whether the system has such a picker: macOS 14 and newer, and Linux under Wayland
- `void start(DesktopCaptureCallback callback)` – begin capturing; frames are delivered via callback. A capturer can be started once, a second call throws `IllegalStateException`
- `void captureFrame()` – request a single frame (manual capture) after `start`; without a capture backend, as on Linux without a display, the callback gets `ERROR_PERMANENT`
- `void dispose()` – release resources
Expand Down Expand Up @@ -71,6 +73,36 @@ finally {
}
```

## Letting the user choose in the system picker

Instead of selecting a source, a capturer can show the operating system's own picker when it starts. On macOS 14 and newer that is ScreenCaptureKit's content sharing picker, which offers a whole screen or one or more windows, for a `ScreenCapturer` and a `WindowCapturer` alike. Under Wayland it is the desktop portal's picker:

```java
ScreenCapturer capturer = new ScreenCapturer();
try {
if (DesktopCapturer.isSystemPickerAvailable()) {
// The picker chooses the source; no selectSource() is needed.
capturer.setUseSystemPicker(true);
}

capturer.start((result, frame) -> {
if (result == DesktopCapturer.Result.SUCCESS && frame != null) {
// process VideoFrame
}
else if (result == DesktopCapturer.Result.ERROR_PERMANENT) {
// the picker was cancelled, or sharing stopped
}
});

// Call captureFrame() periodically
}
finally {
capturer.dispose();
}
```

Set it before `start`, which shows the picker and returns at once. Until the user has chosen, `captureFrame()` reports `ERROR_TEMPORARY`. If the user cancels the picker, or stops sharing from the macOS menu bar, it reports `ERROR_PERMANENT`. Where `isSystemPickerAvailable()` is false, it reports `ERROR_PERMANENT` from the first frame on, instead of capturing something the user did not choose. Show the picker for one capturer at a time: capturers waiting on it at the same time all receive the user's choice.

## Using with VideoDesktopSource to create a VideoTrack

For most WebRTC pipelines, you will create a `VideoDesktopSource` and set the selected screen ID:
Expand Down
6 changes: 6 additions & 0 deletions docs/tools/desktop/window-capturer.md
Original file line number Diff line number Diff line change
Expand Up @@ -16,6 +16,8 @@ Key methods (inherited from `DesktopCapturer`):
- `void selectSource(DesktopSource source)` – choose which window to capture
- `void setFocusSelectedSource(boolean focus)` – attempt to focus/raise the selected window
- `void setMaxFrameRate(int maxFrameRate)` – limit the capture frame rate
- `void setUseSystemPicker(boolean usePicker)` – let the user choose a screen or windows in the operating system's picker, shown by `start`, instead of `selectSource`; set it before `start`, a later call throws `IllegalStateException`
- `static boolean isSystemPickerAvailable()` – whether the system has such a picker: macOS 14 and newer, and Linux under Wayland
- `void start(DesktopCaptureCallback callback)` – start capture, frames via callback. A capturer can be started once, a second call throws `IllegalStateException`
- `void captureFrame()` – request a single capture after `start`; without a capture backend, as on Linux without a display, the callback gets `ERROR_PERMANENT`
- `void dispose()` – release resources
Expand Down Expand Up @@ -68,6 +70,10 @@ finally {
}
```

## Letting the user choose in the system picker

With `setUseSystemPicker(true)` before `start`, the capturer shows the operating system's picker instead of capturing a selected window. It works the same way for a `WindowCapturer` as for a `ScreenCapturer`; see [Letting the user choose in the system picker](/tools/desktop/screen-capturer#letting-the-user-choose-in-the-system-picker).

## Using with VideoDesktopSource to create a VideoTrack

```java
Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,242 @@
/*
* Copyright 2026 Alex Andres
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/

package dev.onvoid.webrtc.examples;

import java.awt.BorderLayout;
import java.awt.Color;
import java.awt.Dimension;
import java.awt.FlowLayout;
import java.awt.event.WindowAdapter;
import java.awt.event.WindowEvent;
import java.util.concurrent.Executors;
import java.util.concurrent.ScheduledExecutorService;
import java.util.concurrent.TimeUnit;

import javax.swing.BorderFactory;
import javax.swing.JButton;
import javax.swing.JFrame;
import javax.swing.JLabel;
import javax.swing.JPanel;
import javax.swing.SwingUtilities;

import dev.onvoid.webrtc.client.swing.VideoView;
import dev.onvoid.webrtc.media.video.VideoFrame;
import dev.onvoid.webrtc.media.video.desktop.DesktopCapturer;
import dev.onvoid.webrtc.media.video.desktop.ScreenCapturer;

/**
* Swing example taking frames from a {@link DesktopCapturer} whose source the
* user chooses in the operating system's own picker, without a video track.
* <p>
* "Choose" starts a {@link ScreenCapturer} set to use the system picker: on
* macOS 14 and newer, ScreenCaptureKit's content sharing picker, which offers
* a screen or one or more windows; under Wayland, the desktop portal's
* picker. The window shows the captured frames and the last capture result.
* Cancelling the picker, or stopping the sharing from the macOS menu bar, ends
* the capture with {@link DesktopCapturer.Result#ERROR_PERMANENT}.
* <p>
* A capturer is used from one thread only, here a single scheduled thread
* that starts it, calls {@link DesktopCapturer#captureFrame()} at the frame
* rate and disposes of it.
* <p>
* Run it with:
* <pre>
* mvn -pl webrtc-examples exec:java -Dexec.mainClass=dev.onvoid.webrtc.examples.DesktopCapturerPickerExample
* </pre>
*
* @author Alex Andres
*/
public class DesktopCapturerPickerExample {

private static final int FRAME_RATE = 30;

private final ScheduledExecutorService captureThread = Executors.newSingleThreadScheduledExecutor(r -> {
Thread thread = new Thread(r, "desktop-capture");
thread.setDaemon(true);
return thread;
});

private final VideoView view = new VideoView();
private final JLabel status = new JLabel();
private final JButton chooseButton = new JButton("Choose…");
private final JButton stopButton = new JButton("Stop");
private final JFrame window = new JFrame("Desktop Capturer Picker");

// Used on the capture thread only.
private DesktopCapturer capturer;
private DesktopCapturer.Result lastResult;
private boolean ended;
private long frames;


public static void main(String[] args) {
SwingUtilities.invokeLater(() -> new DesktopCapturerPickerExample().show());
}

private void show() {
view.setBackground(Color.DARK_GRAY);
view.setPreferredSize(new Dimension(960, 540));

status.setBorder(BorderFactory.createEmptyBorder(0, 8, 0, 8));

chooseButton.addActionListener(e -> startCapture());
stopButton.addActionListener(e -> stopCapture("Stopped."));
stopButton.setEnabled(false);

JPanel controls = new JPanel(new FlowLayout(FlowLayout.LEFT));
controls.add(chooseButton);
controls.add(stopButton);

JPanel bottom = new JPanel(new BorderLayout());
bottom.add(controls, BorderLayout.WEST);
bottom.add(status, BorderLayout.CENTER);

window.setDefaultCloseOperation(JFrame.DISPOSE_ON_CLOSE);
window.addWindowListener(new WindowAdapter() {

@Override
public void windowClosed(WindowEvent e) {
captureThread.execute(() -> disposeCapturer());
captureThread.shutdown();
}
});
window.getContentPane().add(view, BorderLayout.CENTER);
window.getContentPane().add(bottom, BorderLayout.SOUTH);
window.pack();
window.setLocationRelativeTo(null);
window.setVisible(true);

if (DesktopCapturer.isSystemPickerAvailable()) {
setStatus("Press \"Choose…\" to pick a screen or window.");
}
else {
chooseButton.setEnabled(false);
setStatus("This system offers no picker for screens and windows.");
}

// Captures while a capturer is running.
captureThread.scheduleAtFixedRate(this::captureFrame, 0, 1000 / FRAME_RATE, TimeUnit.MILLISECONDS);
}

private void startCapture() {
chooseButton.setEnabled(false);
stopButton.setEnabled(true);
view.clear();

captureThread.execute(() -> {
disposeCapturer();

lastResult = null;
ended = false;
frames = 0;

capturer = new ScreenCapturer();
capturer.setMaxFrameRate(FRAME_RATE);
// Set before start(). No selectSource() is needed: the user
// chooses in the picker, which start() shows.
capturer.setUseSystemPicker(true);
capturer.start(this::onCaptureResult);

setStatus("Choose a screen or window in the picker.");
});
}

private void stopCapture(String message) {
captureThread.execute(() -> {
disposeCapturer();
setStatus(message);
});

chooseButton.setEnabled(true);
stopButton.setEnabled(false);
}

private void captureFrame() {
if (capturer == null) {
return;
}

try {
capturer.captureFrame();
}
catch (RuntimeException e) {
ended = true;
setStatus("Capture failed: " + e.getMessage());
}

if (ended) {
// Disposed of here, not in the callback that reported the end.
disposeCapturer();

SwingUtilities.invokeLater(() -> {
chooseButton.setEnabled(true);
stopButton.setEnabled(false);
});
}
}

/**
* Called on the capture thread from within {@link DesktopCapturer#captureFrame()}.
*/
private void onCaptureResult(DesktopCapturer.Result result, VideoFrame frame) {
try {
switch (result) {
case SUCCESS:
if (frame != null) {
view.accept(frame);
frames++;

setStatus(String.format("Capturing: %d frames, %dx%d", frames,
frame.buffer.getWidth(), frame.buffer.getHeight()));
}
break;

case ERROR_TEMPORARY:
// No frame yet, while the picker waits for the user.
if (lastResult != DesktopCapturer.Result.ERROR_TEMPORARY && frames == 0) {
setStatus("Waiting for a choice in the picker…");
}
break;

default:
ended = true;
setStatus(frames == 0
? "The picker was cancelled."
: "Sharing stopped after " + frames + " frames.");
break;
}

lastResult = result;
}
finally {
if (frame != null) {
frame.release();
}
}
}

private void disposeCapturer() {
if (capturer != null) {
capturer.dispose();
capturer = null;
}
}

private void setStatus(String text) {
SwingUtilities.invokeLater(() -> status.setText(text));
}
}
Loading
Loading