Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions docs/.vitepress/sidebar.ts
Original file line number Diff line number Diff line change
Expand Up @@ -35,6 +35,7 @@ function sidebarGuide(): DefaultTheme.SidebarItem[] {
{ text: 'Audio Devices', link: '/audio/audio-devices' },
{ text: 'Audio Processing', link: '/audio/audio-processing' },
{ text: 'Custom Audio Source', link: '/audio/custom-audio-source' },
{ text: 'System Audio Source', link: '/audio/system-audio-source' },
{ text: 'Headless Audio', link: '/audio/headless-audio' },
{ text: 'DTMF Sender', link: '/audio/dtmf-sender' },
],
Expand Down
116 changes: 116 additions & 0 deletions docs/guide/audio/system-audio-source.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,116 @@
# System Audio Source

`SystemAudioSource` captures audio playing on the system output. It is a
`CustomAudioSource` and can be passed directly to `PeerConnectionFactory.createAudioTrack`.
It sends 16-bit, 48 kHz stereo PCM in 10 ms chunks, independently of screen or
window video capture. Capturing a window does **not** restrict audio to that window.

## Platform support

| Platform | Capture backend | Requirements and scope |
| --- | --- | --- |
| Windows | WASAPI shared-mode loopback | Captures the default multimedia output selected at start, including audio from this process. |
| Linux | PulseAudio output monitor | Requires `libpulse.so.0` and PulseAudio or PipeWire with `pipewire-pulse`. Captures the default sink selected at start, including audio from this process. A native PipeWire portal audio backend is not provided. |
| macOS 13+ | ScreenCaptureKit | Requires screen/audio recording permission and a shareable display. Captures system audio excluding this process. Build with a macOS 13+ SDK. |

No shell commands, virtual audio driver, or FFmpeg installation are required.
There is no application/process selection API in this version. Protected content
and platform recording restrictions still apply.

## Send alongside a screen track

Use your existing factory and peer connection, before negotiation:

```java
import dev.onvoid.webrtc.RTCRtpSender;
import dev.onvoid.webrtc.media.audio.AudioTrack;
import dev.onvoid.webrtc.media.audio.SystemAudioSource;
import java.util.Collections;

SystemAudioSource systemAudio = new SystemAudioSource();
AudioTrack audioTrack = factory.createAudioTrack("presentation-audio", systemAudio);
RTCRtpSender audioSender = peerConnection.addTrack(
audioTrack, Collections.singletonList("presentation"));
// Add your VideoDesktopSource's video track with the same stream ID.
systemAudio.start();
```

`start()` waits for capture to open and throws `IllegalStateException` on startup
failure, with the platform error as its cause. Call it on a background thread,
particularly when macOS may show a permission prompt. Platform startup waits are
bounded; permission dialogs can remain open after a timeout. A late macOS grant
will not leave this timed-out capture recording in the background. Grant permission
in System Settings and retry, restarting the application if the OS requires it.

When the call ends, stop recording before releasing tracks and the source:

```java
systemAudio.stop();
peerConnection.removeTrack(audioSender);
audioSender.dispose();
audioTrack.dispose();
systemAudio.dispose(); // or close(), also supported by try-with-resources
```

Use `try/finally` in application code to release these resources on errors too.
`stop()` and `dispose()` are idempotent. `stop()` keeps the source and its tracks
usable for `start()` again; disposal is permanent. A stop or disposal from a local
audio sink requests shutdown without joining its own callback thread. Do not
restart from an audio callback. The inherited `pushAudio` methods throw
`IllegalStateException`: push other audio into a separate `CustomAudioSource`.

## Device and permission changes

Observe `isRecording()` and `getFailure()` from your application control thread.
A runtime capture failure stops recording and retains its cause in `getFailure()`;
starting again clears that failure. A stopped source does not end its tracks, so
it can resume on the same sender. It does not automatically reopen a device or
silently change recording permissions.

For an output change, call `stop()`, select the new default output through the OS,
and call `start()`. A change of the default output alone does not move a running
capture: Windows and Linux bind the output chosen at startup. Device loss or
permission revocation may only become observable when the platform reports it.

## Synchronization and latency

Capture timestamps use WebRTC's monotonic clock, the same clock as
`SyncClock.currentTimeUs()` and desktop video. WASAPI's packet clock and
ScreenCaptureKit's presentation clock are translated into that clock; PulseAudio
uses its reported capture latency. This preserves the capture time when audio is
buffered before delivery. Queueing is bounded to 100 ms; older chunks are dropped
if delivery stalls. Audio is paced by the OS rather than an independent Java timer.
This does not promise sample-accurate lip synchronization or compensate for all
screen capture, output-device, and receiver latency.

## Microphone and feedback

Send the microphone as a second track of the same factory and connection. Each
sender gets the audio of its own track, so neither track hears the other:

```java
AudioTrackSource microphone = factory.createAudioSource(new AudioOptions());
AudioTrack microphoneTrack = factory.createAudioTrack("microphone", microphone);
peerConnection.addTrack(microphoneTrack, Collections.singletonList("presentation"));
```

The microphone track gets the factory's echo cancellation, noise suppression and
gain control as its `AudioOptions` ask for; the system audio track gets none, and
the two are not mixed. Echo cancellation removes what this application plays, not
what other applications play, so system audio that reaches the microphone through
the speakers is sent twice: use headphones to prevent that.
On Windows and Linux, route received call audio to another output or to a virtual
output that is not monitored. Headphones alone do not prevent a loopback source
from capturing remote audio played on the monitored output. On macOS this process's
playout is excluded by ScreenCaptureKit; remote audio played in another process
can still be captured.

Run `SystemAudioExample` from `webrtc-examples` for a capture-only example that
counts received PCM frames without playing them back. The opt-in
`SystemAudioSourceIntegrationTest` (`-Dtest.systemAudio=true`) needs audio playing
on the default output and recording permission; lifecycle tests use deterministic
backends without hardware.

The platform bindings follow [Microsoft's loopback capture API](https://learn.microsoft.com/en-us/windows/win32/coreaudio/loopback-recording),
[PulseAudio's recording stream API](https://www.freedesktop.org/software/pulseaudio/doxygen/stream_8h.html),
and [ScreenCaptureKit audio configuration](https://developer.apple.com/documentation/screencapturekit/scstreamconfiguration/capturesaudio).
8 changes: 7 additions & 1 deletion docs/guide/video/desktop-capture.md
Original file line number Diff line number Diff line change
Expand Up @@ -189,4 +189,10 @@ When implementing desktop capture in your application, remember to:
- Consider the performance implications of higher resolutions and frame rates
- Handle potential permission requirements on different operating systems

Desktop capture is particularly useful for screen sharing applications, remote assistance tools, collaborative workspaces, and educational platforms where visual content sharing is essential.
Desktop capture is particularly useful for screen sharing applications, remote assistance tools, collaborative workspaces, and educational platforms where visual content sharing is essential.

## Sharing system audio

Add a separate [`SystemAudioSource`](/guide/audio/system-audio-source) audio track
to share output audio alongside desktop video. Window selection applies only to
video; the audio source captures system output independently.
Original file line number Diff line number Diff line change
@@ -0,0 +1,81 @@
/*
* Copyright 2026 Alex Andres
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/

package dev.onvoid.webrtc.examples;

import java.util.concurrent.atomic.AtomicLong;

import dev.onvoid.webrtc.PeerConnectionFactory;
import dev.onvoid.webrtc.media.audio.AudioDeviceModule;
import dev.onvoid.webrtc.media.audio.AudioLayer;
import dev.onvoid.webrtc.media.audio.AudioTrack;
import dev.onvoid.webrtc.media.audio.SystemAudioSource;

/**
* Example demonstrating how to capture the audio the system plays, as a local
* audio track.
* <p>
* This example shows how to:
* <ul>
* <li>Capture the default output with a {@link SystemAudioSource}</li>
* <li>Read the captured audio through a sink of its track</li>
* <li>Stop the capture and check it for a failure</li>
* </ul>
* <p>
* The captured audio is counted, not played back. To send it, add the track to
* a peer connection, beside the track of a desktop video source.
* <p>
* Play audio in another application, run this example, and press Enter to stop:
* <pre>
* mvn exec:java -D"exec.mainClass=dev.onvoid.webrtc.examples.SystemAudioExample"
* </pre>
*
* @author Alex Andres
*/
public class SystemAudioExample {

public static void main(String[] args) throws Exception {
// The source opens its own output capture, so no microphone is needed.
try (AudioDeviceModule audio = new AudioDeviceModule(AudioLayer.kDummyAudio);
PeerConnectionFactory factory = new PeerConnectionFactory(audio);
SystemAudioSource source = new SystemAudioSource()) {
AudioTrack track = factory.createAudioTrack("system-audio", source);
AtomicLong frames = new AtomicLong();

try {
track.addSink((data, bits, rate, channels, count) -> frames.addAndGet(count));

source.start();

System.out.println("Capturing system audio. Play audio in another application; press Enter to stop.");
System.in.read();

source.stop();

if (source.getFailure() != null) {
throw new IllegalStateException("System audio capture failed", source.getFailure());
}

System.out.println("Captured " + frames.get() + " stereo PCM frames");
}
finally {
source.stop();
track.dispose();
}
}
}

}
2 changes: 1 addition & 1 deletion webrtc-jni/src/main/cpp/CMakeLists.txt
Original file line number Diff line number Diff line change
Expand Up @@ -139,7 +139,7 @@ target_link_libraries(${PROJECT_NAME} webrtc)
if(APPLE)
set_source_files_properties(${SOURCES} PROPERTIES COMPILE_FLAGS "-x objective-c++")
target_link_options(${PROJECT_NAME} PRIVATE "-ObjC")
target_link_libraries(${PROJECT_NAME} "-framework Foundation" "-framework AVFoundation" "-framework CoreMedia" "-framework CoreAudio" "-framework IOKit" "-framework CoreVideo" "-framework VideoToolbox" "-framework QuartzCore")
target_link_libraries(${PROJECT_NAME} "-weak_framework ScreenCaptureKit" "-framework Foundation" "-framework AVFoundation" "-framework CoreMedia" "-framework CoreAudio" "-framework IOKit" "-framework CoreVideo" "-framework VideoToolbox" "-framework QuartzCore")
elseif(LINUX)
set(CXX_LIBS "-static-libgcc -stdlib=libc++ -lc++ -lc++abi")

Expand Down
37 changes: 37 additions & 0 deletions webrtc-jni/src/main/cpp/include/JNI_SystemAudioSource.h

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

86 changes: 86 additions & 0 deletions webrtc-jni/src/main/cpp/include/media/audio/SystemAudioCapture.h
Original file line number Diff line number Diff line change
@@ -0,0 +1,86 @@
/*
* Copyright 2026 Alex Andres
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/

#ifndef JNI_WEBRTC_MEDIA_AUDIO_SYSTEM_AUDIO_CAPTURE_H_
#define JNI_WEBRTC_MEDIA_AUDIO_SYSTEM_AUDIO_CAPTURE_H_

#include "media/audio/CustomAudioSource.h"

#include <array>
#include <cstddef>
#include <cstdint>
#include <deque>
#include <memory>
#include <mutex>
#include <string>

namespace jni
{
// Captures what the system plays on its default output, as 16-bit 48 kHz
// stereo PCM. Each platform implements Open() and Poll().
//
// Platform callbacks only write PCM into a bounded queue; they never call
// Java or WebRTC. The owning Java worker calls Pump(), which polls the
// platform and delivers the complete 10 ms chunks to a CustomAudioSource.
// Open(), Pump() and the destructor run on that one worker thread.
class SystemAudioCapture
{
public:
static constexpr int kSampleRate = 48000;
static constexpr size_t kChannels = 2;
static constexpr size_t kFramesPerChunk = kSampleRate / 100;

// Creates the capture of the current platform, not opened yet.
static std::unique_ptr<SystemAudioCapture> Create();

virtual ~SystemAudioCapture() = default;

// Opens the default output. Throws an Exception on failure.
virtual void Open() = 0;

// Appends the audio the platform has buffered since the last call.
// Throws an Exception once capture has failed.
virtual void Poll() = 0;

// Polls the platform and pushes every complete chunk to the source,
// stamped with its capture time.
void Pump(CustomAudioSource * source);

// Queues interleaved frames whose first one was captured at the given
// time, on the clock of webrtc::TimeMicros(). Null samples queue
// silence of the given length.
void Append(const int16_t * samples, size_t frames, int64_t timestamp_us);

// Records a failure, which the next Pump() throws. Safe to call from
// platform callbacks.
void Fail(const char * message) noexcept;

private:
struct Chunk
{
std::array<int16_t, kFramesPerChunk * kChannels> samples{};
size_t frames = 0;
int64_t timestamp_us = 0;
};

// Guards chunks_ and failure_.
std::mutex mutex_;
std::deque<Chunk> chunks_;
std::string failure_;
};
}

#endif
Loading
Loading