diff --git a/docs/.vitepress/sidebar.ts b/docs/.vitepress/sidebar.ts index 9d7b3d913..6613c3863 100644 --- a/docs/.vitepress/sidebar.ts +++ b/docs/.vitepress/sidebar.ts @@ -35,6 +35,7 @@ function sidebarGuide(): DefaultTheme.SidebarItem[] { { text: 'Audio Devices', link: '/audio/audio-devices' }, { text: 'Audio Processing', link: '/audio/audio-processing' }, { text: 'Custom Audio Source', link: '/audio/custom-audio-source' }, + { text: 'System Audio Source', link: '/audio/system-audio-source' }, { text: 'Headless Audio', link: '/audio/headless-audio' }, { text: 'DTMF Sender', link: '/audio/dtmf-sender' }, ], diff --git a/docs/guide/audio/system-audio-source.md b/docs/guide/audio/system-audio-source.md new file mode 100644 index 000000000..295f710ab --- /dev/null +++ b/docs/guide/audio/system-audio-source.md @@ -0,0 +1,116 @@ +# System Audio Source + +`SystemAudioSource` captures audio playing on the system output. It is a +`CustomAudioSource` and can be passed directly to `PeerConnectionFactory.createAudioTrack`. +It sends 16-bit, 48 kHz stereo PCM in 10 ms chunks, independently of screen or +window video capture. Capturing a window does **not** restrict audio to that window. + +## Platform support + +| Platform | Capture backend | Requirements and scope | +| --- | --- | --- | +| Windows | WASAPI shared-mode loopback | Captures the default multimedia output selected at start, including audio from this process. | +| Linux | PulseAudio output monitor | Requires `libpulse.so.0` and PulseAudio or PipeWire with `pipewire-pulse`. Captures the default sink selected at start, including audio from this process. A native PipeWire portal audio backend is not provided. | +| macOS 13+ | ScreenCaptureKit | Requires screen/audio recording permission and a shareable display. Captures system audio excluding this process. Build with a macOS 13+ SDK. | + +No shell commands, virtual audio driver, or FFmpeg installation are required. +There is no application/process selection API in this version. Protected content +and platform recording restrictions still apply. + +## Send alongside a screen track + +Use your existing factory and peer connection, before negotiation: + +```java +import dev.onvoid.webrtc.RTCRtpSender; +import dev.onvoid.webrtc.media.audio.AudioTrack; +import dev.onvoid.webrtc.media.audio.SystemAudioSource; +import java.util.Collections; + +SystemAudioSource systemAudio = new SystemAudioSource(); +AudioTrack audioTrack = factory.createAudioTrack("presentation-audio", systemAudio); +RTCRtpSender audioSender = peerConnection.addTrack( + audioTrack, Collections.singletonList("presentation")); +// Add your VideoDesktopSource's video track with the same stream ID. +systemAudio.start(); +``` + +`start()` waits for capture to open and throws `IllegalStateException` on startup +failure, with the platform error as its cause. Call it on a background thread, +particularly when macOS may show a permission prompt. Platform startup waits are +bounded; permission dialogs can remain open after a timeout. A late macOS grant +will not leave this timed-out capture recording in the background. Grant permission +in System Settings and retry, restarting the application if the OS requires it. + +When the call ends, stop recording before releasing tracks and the source: + +```java +systemAudio.stop(); +peerConnection.removeTrack(audioSender); +audioSender.dispose(); +audioTrack.dispose(); +systemAudio.dispose(); // or close(), also supported by try-with-resources +``` + +Use `try/finally` in application code to release these resources on errors too. +`stop()` and `dispose()` are idempotent. `stop()` keeps the source and its tracks +usable for `start()` again; disposal is permanent. A stop or disposal from a local +audio sink requests shutdown without joining its own callback thread. Do not +restart from an audio callback. The inherited `pushAudio` methods throw +`IllegalStateException`: push other audio into a separate `CustomAudioSource`. + +## Device and permission changes + +Observe `isRecording()` and `getFailure()` from your application control thread. +A runtime capture failure stops recording and retains its cause in `getFailure()`; +starting again clears that failure. A stopped source does not end its tracks, so +it can resume on the same sender. It does not automatically reopen a device or +silently change recording permissions. + +For an output change, call `stop()`, select the new default output through the OS, +and call `start()`. A change of the default output alone does not move a running +capture: Windows and Linux bind the output chosen at startup. Device loss or +permission revocation may only become observable when the platform reports it. + +## Synchronization and latency + +Capture timestamps use WebRTC's monotonic clock, the same clock as +`SyncClock.currentTimeUs()` and desktop video. WASAPI's packet clock and +ScreenCaptureKit's presentation clock are translated into that clock; PulseAudio +uses its reported capture latency. This preserves the capture time when audio is +buffered before delivery. Queueing is bounded to 100 ms; older chunks are dropped +if delivery stalls. Audio is paced by the OS rather than an independent Java timer. +This does not promise sample-accurate lip synchronization or compensate for all +screen capture, output-device, and receiver latency. + +## Microphone and feedback + +Send the microphone as a second track of the same factory and connection. Each +sender gets the audio of its own track, so neither track hears the other: + +```java +AudioTrackSource microphone = factory.createAudioSource(new AudioOptions()); +AudioTrack microphoneTrack = factory.createAudioTrack("microphone", microphone); +peerConnection.addTrack(microphoneTrack, Collections.singletonList("presentation")); +``` + +The microphone track gets the factory's echo cancellation, noise suppression and +gain control as its `AudioOptions` ask for; the system audio track gets none, and +the two are not mixed. Echo cancellation removes what this application plays, not +what other applications play, so system audio that reaches the microphone through +the speakers is sent twice: use headphones to prevent that. +On Windows and Linux, route received call audio to another output or to a virtual +output that is not monitored. Headphones alone do not prevent a loopback source +from capturing remote audio played on the monitored output. On macOS this process's +playout is excluded by ScreenCaptureKit; remote audio played in another process +can still be captured. + +Run `SystemAudioExample` from `webrtc-examples` for a capture-only example that +counts received PCM frames without playing them back. The opt-in +`SystemAudioSourceIntegrationTest` (`-Dtest.systemAudio=true`) needs audio playing +on the default output and recording permission; lifecycle tests use deterministic +backends without hardware. + +The platform bindings follow [Microsoft's loopback capture API](https://learn.microsoft.com/en-us/windows/win32/coreaudio/loopback-recording), +[PulseAudio's recording stream API](https://www.freedesktop.org/software/pulseaudio/doxygen/stream_8h.html), +and [ScreenCaptureKit audio configuration](https://developer.apple.com/documentation/screencapturekit/scstreamconfiguration/capturesaudio). diff --git a/docs/guide/video/desktop-capture.md b/docs/guide/video/desktop-capture.md index 4207121b9..da9127079 100644 --- a/docs/guide/video/desktop-capture.md +++ b/docs/guide/video/desktop-capture.md @@ -189,4 +189,10 @@ When implementing desktop capture in your application, remember to: - Consider the performance implications of higher resolutions and frame rates - Handle potential permission requirements on different operating systems -Desktop capture is particularly useful for screen sharing applications, remote assistance tools, collaborative workspaces, and educational platforms where visual content sharing is essential. \ No newline at end of file +Desktop capture is particularly useful for screen sharing applications, remote assistance tools, collaborative workspaces, and educational platforms where visual content sharing is essential. + +## Sharing system audio + +Add a separate [`SystemAudioSource`](/guide/audio/system-audio-source) audio track +to share output audio alongside desktop video. Window selection applies only to +video; the audio source captures system output independently. diff --git a/webrtc-examples/src/main/java/dev/onvoid/webrtc/examples/SystemAudioExample.java b/webrtc-examples/src/main/java/dev/onvoid/webrtc/examples/SystemAudioExample.java new file mode 100644 index 000000000..e0ea8e01f --- /dev/null +++ b/webrtc-examples/src/main/java/dev/onvoid/webrtc/examples/SystemAudioExample.java @@ -0,0 +1,81 @@ +/* + * Copyright 2026 Alex Andres + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package dev.onvoid.webrtc.examples; + +import java.util.concurrent.atomic.AtomicLong; + +import dev.onvoid.webrtc.PeerConnectionFactory; +import dev.onvoid.webrtc.media.audio.AudioDeviceModule; +import dev.onvoid.webrtc.media.audio.AudioLayer; +import dev.onvoid.webrtc.media.audio.AudioTrack; +import dev.onvoid.webrtc.media.audio.SystemAudioSource; + +/** + * Example demonstrating how to capture the audio the system plays, as a local + * audio track. + *
+ * This example shows how to: + *
+ * The captured audio is counted, not played back. To send it, add the track to + * a peer connection, beside the track of a desktop video source. + *
+ * Play audio in another application, run this example, and press Enter to stop: + *
+ * mvn exec:java -D"exec.mainClass=dev.onvoid.webrtc.examples.SystemAudioExample" + *+ * + * @author Alex Andres + */ +public class SystemAudioExample { + + public static void main(String[] args) throws Exception { + // The source opens its own output capture, so no microphone is needed. + try (AudioDeviceModule audio = new AudioDeviceModule(AudioLayer.kDummyAudio); + PeerConnectionFactory factory = new PeerConnectionFactory(audio); + SystemAudioSource source = new SystemAudioSource()) { + AudioTrack track = factory.createAudioTrack("system-audio", source); + AtomicLong frames = new AtomicLong(); + + try { + track.addSink((data, bits, rate, channels, count) -> frames.addAndGet(count)); + + source.start(); + + System.out.println("Capturing system audio. Play audio in another application; press Enter to stop."); + System.in.read(); + + source.stop(); + + if (source.getFailure() != null) { + throw new IllegalStateException("System audio capture failed", source.getFailure()); + } + + System.out.println("Captured " + frames.get() + " stereo PCM frames"); + } + finally { + source.stop(); + track.dispose(); + } + } + } + +} diff --git a/webrtc-jni/src/main/cpp/CMakeLists.txt b/webrtc-jni/src/main/cpp/CMakeLists.txt index 1befe80b6..29a50daa6 100644 --- a/webrtc-jni/src/main/cpp/CMakeLists.txt +++ b/webrtc-jni/src/main/cpp/CMakeLists.txt @@ -139,7 +139,7 @@ target_link_libraries(${PROJECT_NAME} webrtc) if(APPLE) set_source_files_properties(${SOURCES} PROPERTIES COMPILE_FLAGS "-x objective-c++") target_link_options(${PROJECT_NAME} PRIVATE "-ObjC") - target_link_libraries(${PROJECT_NAME} "-framework Foundation" "-framework AVFoundation" "-framework CoreMedia" "-framework CoreAudio" "-framework IOKit" "-framework CoreVideo" "-framework VideoToolbox" "-framework QuartzCore") + target_link_libraries(${PROJECT_NAME} "-weak_framework ScreenCaptureKit" "-framework Foundation" "-framework AVFoundation" "-framework CoreMedia" "-framework CoreAudio" "-framework IOKit" "-framework CoreVideo" "-framework VideoToolbox" "-framework QuartzCore") elseif(LINUX) set(CXX_LIBS "-static-libgcc -stdlib=libc++ -lc++ -lc++abi") diff --git a/webrtc-jni/src/main/cpp/include/JNI_SystemAudioSource.h b/webrtc-jni/src/main/cpp/include/JNI_SystemAudioSource.h new file mode 100644 index 000000000..6e81ab834 --- /dev/null +++ b/webrtc-jni/src/main/cpp/include/JNI_SystemAudioSource.h @@ -0,0 +1,37 @@ +/* DO NOT EDIT THIS FILE - it is machine generated */ +#include
+ * Windows uses WASAPI loopback of the default multimedia output; Linux uses the + * default output's PulseAudio monitor (also available with pipewire-pulse). + * macOS 13 or later uses ScreenCaptureKit and excludes this process's audio. + * Screen/audio recording permission is required on macOS. This class does not + * capture a selected window's or process's audio. + *
+ * Call {@link #start()} to begin and {@link #stop()} before changing output
+ * devices. A restart selects the current default output. Capture errors stop
+ * recording and are available through {@link #getFailure()}. Remote call audio
+ * played on the captured output is included on Windows and Linux: route it to
+ * another output to prevent feedback. No microphone mixing or echo cancellation
+ * is performed. The inherited {@code pushAudio} methods are rejected, since the
+ * captured audio is the only audio this source sends.
+ *
+ * @author Alex Andres
+ */
+public final class SystemAudioSource extends CustomAudioSource implements AutoCloseable {
+
+ /**
+ * How long the capture worker waits between two polls of the platform. A
+ * chunk lasts 10 ms, and the platform buffers up to 100 ms.
+ */
+ private static final long POLL_INTERVAL_NANOS = 2_000_000L;
+
+ /** Creates the platform capture, once for each start. */
+ private final Supplier