Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion docs/guide/audio/audio-processing.md
Original file line number Diff line number Diff line change
Expand Up @@ -300,7 +300,7 @@ audioProcessing.processStream(

## Speech in the Processed Microphone Audio

WebRTC passes the microphone's audio from the audio device module straight to the senders, so a sink on a microphone track gets nothing. The factory detects speech in that audio as it is sent, after echo cancellation, noise suppression and gain control, in native code, and tells a listener only when speech begins and ends:
A sink on a microphone track gets the audio its senders send, after echo cancellation, noise suppression and gain control, on WebRTC's capture thread. To learn only when speech begins and ends, without handling the audio in Java, let the factory detect speech in that audio in native code and tell a listener:

```java
VoiceActivityListener listener = speaking -> {
Expand Down
21 changes: 21 additions & 0 deletions webrtc-jni/src/main/cpp/include/JNI_ToneAudioDeviceModule.h

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

28 changes: 26 additions & 2 deletions webrtc-jni/src/main/cpp/include/api/ProxyAudioDeviceModule.h
Original file line number Diff line number Diff line change
Expand Up @@ -19,11 +19,15 @@

#include "api/audio/audio_device.h"
#include "api/audio/audio_device_defines.h"
#include "api/audio/audio_processing.h"
#include "api/scoped_refptr.h"
#include "rtc_base/thread.h"

#include "media/audio/DeviceAudioSource.h"

#include <atomic>
#include <cstdint>
#include <memory>
#include <optional>

namespace jni
Expand All @@ -41,6 +45,14 @@ namespace jni
// AudioSendStream::SendAudioData and aborts the process, and interleaves
// microphone frames with the pushed audio whenever it does not.
//
// Recorded audio therefore never reaches WebRTC's own transport. While
// capture is enabled the proxy hands it to its DeviceAudioCapture, which
// runs it through the factory's AudioProcessing and on to the
// DeviceAudioSources, the sources PeerConnectionFactory.createAudioSource()
// creates. Their tracks' senders receive it through the tracks' sinks, like
// the senders of every other local source. Render calls still go to
// WebRTC, which feeds the same AudioProcessing the playout signal.
//
// While capture is disabled, InitRecording() and StartRecording() fail so
// AudioState never opens the device, and recorded data that still arrives
// from the wrapped module (from a recording the application started through
Expand Down Expand Up @@ -71,6 +83,14 @@ namespace jni
void SetCaptureEnabled(bool enabled);
bool CaptureEnabled() const;

// Sets the AudioProcessing the factory is built with, which the
// recorded audio runs through. Called before the factory exists.
void SetAudioProcessing(webrtc::scoped_refptr<webrtc::AudioProcessing> processing);

// The capture the factory's DeviceAudioSources receive the recorded
// audio from. The sources share it, and may outlive this proxy.
std::shared_ptr<DeviceAudioCapture> DeviceCapture() const;

// webrtc::AudioDeviceModule implementation.
int32_t ActiveAudioLayer(AudioLayer * audioLayer) const override;
int32_t RegisterAudioCallback(webrtc::AudioTransport * audioCallback) override;
Expand Down Expand Up @@ -154,11 +174,12 @@ namespace jni

// The AudioTransport registered with the wrapped module in place of
// the one WebRTC registers with this proxy. Render calls are always
// forwarded; recorded data is forwarded only while capture is enabled.
// forwarded to WebRTC; recorded data goes to the DeviceAudioCapture,
// and only while capture is enabled.
class TransportGate : public webrtc::AudioTransport
{
public:
explicit TransportGate(const std::atomic<bool> & captureEnabled);
TransportGate(const std::atomic<bool> & captureEnabled, DeviceAudioCapture & capture);
~TransportGate() override = default;

void SetTarget(webrtc::AudioTransport * target);
Expand Down Expand Up @@ -203,11 +224,14 @@ namespace jni

private:
const std::atomic<bool> & captureEnabled_;
DeviceAudioCapture & capture_;
std::atomic<webrtc::AudioTransport *> target_;
};

webrtc::scoped_refptr<webrtc::AudioDeviceModule> delegate_;
std::atomic<bool> captureEnabled_;
// Declared before the gate, which refers to it.
const std::shared_ptr<DeviceAudioCapture> deviceCapture_;
TransportGate gate_;
bool gateRegistered_;
// The factory's worker thread, set while WebRTC's transport is registered.
Expand Down
68 changes: 68 additions & 0 deletions webrtc-jni/src/main/cpp/include/api/ToneAudioDeviceModule.h
Original file line number Diff line number Diff line change
@@ -0,0 +1,68 @@
/*
* Copyright 2026 Alex Andres
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/

#ifndef JNI_WEBRTC_API_TONE_AUDIO_DEVICE_MODULE_H_
#define JNI_WEBRTC_API_TONE_AUDIO_DEVICE_MODULE_H_

#include "api/HeadlessAudioDeviceModule.h"

#include "rtc_base/platform_thread.h"
#include "rtc_base/synchronization/mutex.h"

#include <cstdint>

namespace jni
{
// A HeadlessAudioDeviceModule that also records: while recording, it hands
// its transport a sine tone in real time, 10 ms at a time, the way a device
// module hands it a microphone. Only the tests reach it, so that they can
// send device-captured audio without audio hardware.
class ToneAudioDeviceModule : public HeadlessAudioDeviceModule
{
public:
static webrtc::scoped_refptr<ToneAudioDeviceModule> Create(int frequencyHz)
{
return webrtc::make_ref_counted<ToneAudioDeviceModule>(frequencyHz);
}

explicit ToneAudioDeviceModule(int frequencyHz);
~ToneAudioDeviceModule() override;

int32_t RegisterAudioCallback(webrtc::AudioTransport * audioCallback) override;
int32_t Terminate() override;
int32_t StartRecording() override;
int32_t StopRecording() override;

private:
static constexpr int kSampleRate = 48000;
static constexpr size_t kFramesPer10Ms = kSampleRate / 100;

// Records 10 ms; returns false once recording stops.
bool RecordThreadProcess();

const int frequencyHz_;

webrtc::Mutex recordMutex_;
bool recordRunning_ RTC_GUARDED_BY(recordMutex_) = false;
webrtc::AudioTransport * recordCallback_ RTC_GUARDED_BY(recordMutex_) = nullptr;
int64_t nextRecordMillis_ = 0;
uint64_t phase_ = 0;

webrtc::PlatformThread recordThread_;
};
}

#endif
128 changes: 128 additions & 0 deletions webrtc-jni/src/main/cpp/include/media/audio/DeviceAudioSource.h
Original file line number Diff line number Diff line change
@@ -0,0 +1,128 @@
/*
* Copyright 2026 Alex Andres
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/

#ifndef JNI_WEBRTC_MEDIA_AUDIO_DEVICE_AUDIO_SOURCE_H_
#define JNI_WEBRTC_MEDIA_AUDIO_DEVICE_AUDIO_SOURCE_H_

#include "api/audio/audio_frame.h"
#include "api/audio/audio_processing.h"
#include "api/audio_options.h"
#include "api/media_stream_interface.h"
#include "api/notifier.h"
#include "api/scoped_refptr.h"
#include "common_audio/resampler/include/push_resampler.h"
#include "rtc_base/synchronization/mutex.h"
#include "rtc_base/thread_annotations.h"

#include <cstddef>
#include <cstdint>
#include <memory>
#include <optional>
#include <vector>

namespace jni
{
class DeviceAudioSource;

// Hands the audio recorded by a factory's audio device module to the
// factory's DeviceAudioSources, after running it through the factory's
// AudioProcessing.
//
// WebRTC would otherwise process the recorded audio itself and feed it into
// every audio send stream of the factory, whatever the stream's track. Here
// it reaches only the senders of device-captured tracks, through the tracks'
// sinks, like the audio of every other kind of local source. The playout
// side is left to WebRTC, which feeds the same AudioProcessing the render
// signal, so echo cancellation sees the audio it has to remove.
//
// Shared by the factory's ProxyAudioDeviceModule, which feeds it on the
// recording thread, and by every DeviceAudioSource it created, which may
// outlive the module.
class DeviceAudioCapture
{
public:
DeviceAudioCapture() = default;
DeviceAudioCapture(const DeviceAudioCapture &) = delete;
DeviceAudioCapture & operator=(const DeviceAudioCapture &) = delete;

// Sets the AudioProcessing the factory was built with. Called once,
// before the factory exists.
void SetAudioProcessing(webrtc::scoped_refptr<webrtc::AudioProcessing> processing);

// Releases the AudioProcessing and every source, once the module is
// gone and no more audio can arrive.
void Detach();

// Called on the recording thread with 10 ms of interleaved 16-bit
// audio. Processes it and hands it to the sources that have sinks.
void OnRecordedData(const void * audioSamples, size_t frames, size_t bytesPerSample,
size_t channels, uint32_t sampleRate, uint32_t totalDelayMs,
bool keyPressed, std::optional<int64_t> estimatedCaptureTimeNs);

// A source is registered while it has at least one sink, and held
// for that long, so it is alive whenever audio is handed to it.
void AddSource(webrtc::scoped_refptr<DeviceAudioSource> source);
webrtc::scoped_refptr<DeviceAudioSource> RemoveSource(DeviceAudioSource * source);

private:
// Guards the processing state, which only the recording thread
// uses; it changes threads when the device is restarted.
webrtc::Mutex processMutex_;
webrtc::scoped_refptr<webrtc::AudioProcessing> processing_ RTC_GUARDED_BY(processMutex_);
webrtc::AudioFrame frame_ RTC_GUARDED_BY(processMutex_);
webrtc::PushResampler<int16_t> resampler_ RTC_GUARDED_BY(processMutex_);

// Not held while the sources deliver: the list is copied first, so
// adding or removing a sink never waits for another source's sinks.
webrtc::Mutex sourcesMutex_;
std::vector<webrtc::scoped_refptr<DeviceAudioSource>> sources_ RTC_GUARDED_BY(sourcesMutex_);
};

// The source of a track that sends audio captured by the factory's audio
// device module, as PeerConnectionFactory.createAudioSource() creates it.
// Its sinks, the senders of its tracks among them, receive the recorded
// audio after the factory's AudioProcessing, on the recording thread.
class DeviceAudioSource : public webrtc::Notifier<webrtc::AudioSourceInterface>
{
public:
DeviceAudioSource(const webrtc::AudioOptions & options, std::shared_ptr<DeviceAudioCapture> capture);
~DeviceAudioSource() override;

// webrtc::MediaSourceInterface implementation.
SourceState state() const override;
bool remote() const override;

// webrtc::AudioSourceInterface implementation. WebRTC applies the
// options to the factory's AudioProcessing when a sender starts.
const webrtc::AudioOptions options() const override;
void AddSink(webrtc::AudioTrackSinkInterface * sink) override;
void RemoveSink(webrtc::AudioTrackSinkInterface * sink) override;

// Hands one processed frame to every sink.
void Deliver(const webrtc::AudioFrame & frame, std::optional<int64_t> absoluteCaptureTimeMs);

private:
const webrtc::AudioOptions options_;
const std::shared_ptr<DeviceAudioCapture> capture_;

// Held for a whole delivery, as CustomAudioSource does, so that a
// sink is not called once RemoveSink() has returned.
webrtc::Mutex mutex_;
std::vector<webrtc::AudioTrackSinkInterface *> sinks_ RTC_GUARDED_BY(mutex_);
};
}

#endif
42 changes: 30 additions & 12 deletions webrtc-jni/src/main/cpp/src/JNI_PeerConnectionFactory.cpp
Original file line number Diff line number Diff line change
Expand Up @@ -47,11 +47,13 @@

#include "api/ProxyAudioDeviceModule.h"
#include "media/audio/CustomAudioSource.h"
#include "media/audio/DeviceAudioSource.h"
#include "media/audio/codec/AudioCodecFactories.h"
#include "media/video/codec/DefaultVideoCodecFactories.h"
#include "media/video/codec/VideoDecoderFactoryWrapper.h"
#include "media/video/codec/VideoEncoderFactoryWrapper.h"

#include "api/make_ref_counted.h"
#include "api/media_stream_interface.h"
#include "rtc_base/logging.h"
#include "rtc_base/thread.h"
Expand Down Expand Up @@ -159,14 +161,21 @@ JNIEXPORT void JNICALL Java_dev_onvoid_webrtc_PeerConnectionFactory_initialize
dependencies.signaling_thread = signalingThread.get();
dependencies.socket_factory = networkThread->socketserver();
dependencies.event_log_factory = std::make_unique<webrtc::RtcEventLogFactory>();
dependencies.env = webrtc::CreateEnvironment(std::move(fieldTrials));

webrtc::Environment environment = webrtc::CreateEnvironment(std::move(fieldTrials));
dependencies.env = environment;
dependencies.adm = proxy;
dependencies.audio_encoder_factory = audioEncoderFactory;
dependencies.audio_decoder_factory = audioDecoderFactory;
// The tap of the module that processes the captured audio: that of the
// application's module, or of the one built here.
std::shared_ptr<jni::AudioCaptureTap> tap;

// The module is built here rather than by the factory, since the proxy
// runs the recorded audio through it and WebRTC the playout audio: both
// must reach the same module for echo cancellation to work.
webrtc::scoped_refptr<webrtc::AudioProcessing> audioProcessingModule;

if (processing != nullptr) {
auto moduleTap = GetHandle<std::shared_ptr<jni::AudioCaptureTap>>(env, audioProcessing, "captureTapHandle");

Expand All @@ -175,17 +184,24 @@ JNIEXPORT void JNICALL Java_dev_onvoid_webrtc_PeerConnectionFactory_initialize
}

tap = *moduleTap;
dependencies.audio_processing_builder = webrtc::CustomAudioProcessing(
webrtc::scoped_refptr<webrtc::AudioProcessing>(processing));
audioProcessingModule = webrtc::scoped_refptr<webrtc::AudioProcessing>(processing);
}
else {
tap = std::make_shared<jni::AudioCaptureTap>();

auto builder = std::make_unique<webrtc::BuiltinAudioProcessingBuilder>();
builder->SetCapturePostProcessing(jni::AudioCaptureTap::CreateProcessing(tap));
webrtc::BuiltinAudioProcessingBuilder builder;
builder.SetCapturePostProcessing(jni::AudioCaptureTap::CreateProcessing(tap));

audioProcessingModule = builder.Build(environment);

dependencies.audio_processing_builder = std::move(builder);
if (!audioProcessingModule) {
throw jni::Exception("Create AudioProcessing failed");
}
}

proxy->SetAudioProcessing(audioProcessingModule);

dependencies.audio_processing_builder = webrtc::CustomAudioProcessing(audioProcessingModule);
dependencies.video_encoder_factory = std::move(videoEncoderFactory);
dependencies.video_decoder_factory = std::move(videoDecoderFactory);

Expand Down Expand Up @@ -352,14 +368,16 @@ JNIEXPORT jobject JNICALL Java_dev_onvoid_webrtc_PeerConnectionFactory_createAud
jni::NativeLease<webrtc::PeerConnectionFactoryInterface> factory(env, caller);
CHECK_HANDLEV(factory, nullptr);

auto audioOptions = jni::AudioOptions::toNative(env, jni::JavaLocalRef<jobject>(env, jAudioOptions));
jni::ProxyAudioDeviceModule * audioModule = GetHandle<jni::ProxyAudioDeviceModule>(env, caller, "audioModuleHandle");
CHECK_HANDLEV(audioModule, nullptr);

webrtc::scoped_refptr<webrtc::AudioSourceInterface> audioSource = factory->CreateAudioSource(audioOptions);
auto audioOptions = jni::AudioOptions::toNative(env, jni::JavaLocalRef<jobject>(env, jAudioOptions));

if (audioSource == nullptr) {
env->Throw(jni::JavaError(env, "Create audio-track source failed"));
return nullptr;
}
// Not WebRTC's own LocalAudioSource, which leaves the recorded audio to
// reach the senders past the track: this source hands it to the track's
// sinks, its senders among them. See DeviceAudioCapture.
webrtc::scoped_refptr<webrtc::AudioSourceInterface> audioSource =
webrtc::make_ref_counted<jni::DeviceAudioSource>(audioOptions, audioModule->DeviceCapture());

return jni::JavaFactories::createOwning(env, std::move(audioSource)).release();
}
Expand Down
Loading
Loading