From d338df92996386980db835cf69639d86661312d7 Mon Sep 17 00:00:00 2001 From: Alex Andres Date: Sun, 8 Nov 2026 14:52:20 +0100 Subject: [PATCH] feat: send device-captured and pushed audio from one factory Each audio sender now gets the audio of its own track, so the factory no longer commits to one kind of audio input. The proxy's capture opens the recording device only while a device source has a sink and WebRTC asks for a recording; WebRTC's own recording calls no longer reach the device. The client API allows customAudio() in AudioMode.DEVICES. --- docs/guide/audio/custom-audio-source.md | 13 +- docs/guide/client-api-design.md | 2 +- docs/guide/client-api.md | 10 +- docs/guide/media/media-files.md | 9 +- .../cpp/include/JNI_PeerConnectionFactory.h | 16 -- .../cpp/include/JNI_ToneAudioDeviceModule.h | 8 + .../cpp/include/api/ProxyAudioDeviceModule.h | 56 ++-- .../include/media/audio/DeviceAudioSource.h | 60 ++++- .../cpp/src/JNI_PeerConnectionFactory.cpp | 58 +--- .../cpp/src/JNI_ToneAudioDeviceModule.cpp | 15 ++ .../cpp/src/api/ProxyAudioDeviceModule.cpp | 113 +++----- .../cpp/src/media/audio/DeviceAudioSource.cpp | 140 ++++++++++ webrtc-media/README.md | 4 +- .../webrtc/media/player/MediaFileSource.java | 5 +- .../webrtc/media/player/MediaPlayer.java | 5 +- .../onvoid/webrtc/PeerConnectionFactory.java | 147 +--------- .../dev/onvoid/webrtc/RTCPeerConnection.java | 52 +--- .../dev/onvoid/webrtc/client/AudioMode.java | 17 +- .../dev/onvoid/webrtc/client/LocalMedia.java | 8 +- .../onvoid/webrtc/client/LocalMediaImpl.java | 14 +- .../onvoid/webrtc/DeviceAudioSendTests.java | 252 ++++++++++++++++-- .../webrtc/PeerConnectionFactoryTests.java | 175 +++--------- .../media/audio/ToneAudioDeviceModule.java | 8 + 23 files changed, 605 insertions(+), 582 deletions(-) diff --git a/docs/guide/audio/custom-audio-source.md b/docs/guide/audio/custom-audio-source.md index 333e4e42b..01ad7c764 100644 --- a/docs/guide/audio/custom-audio-source.md +++ b/docs/guide/audio/custom-audio-source.md @@ -45,17 +45,10 @@ PeerConnectionFactory factory = new PeerConnectionFactory(); AudioTrack audioTrack = factory.createAudioTrack("audio-track-id", audioSource); ``` -::: warning One kind of audio input per factory -A factory sends audio either from its `AudioDeviceModule`, which is what tracks created with `createAudioSource(AudioOptions)` send, or from audio you push. WebRTC feeds device-captured audio into **every** audio sender of a factory, so a sender that is fed both ways is fed by two threads at once. That is a race inside WebRTC, and it aborts the whole process: +::: tip Pushed and device-captured audio side by side +Each audio sender gets the audio of its own track, so one factory, and one peer connection, can send a `CustomAudioSource` track beside a microphone track created with `createAudioSource(AudioOptions)`, and beside a track forwarded from a remote peer. -``` -Fatal error in: ../../audio/audio_send_stream.cc -Check failed: !race_checker.RaceDetected() -``` - -The factory therefore commits to the kind used first and throws an `IllegalStateException` if you ask for the other. The same applies to a track received from a remote peer and forwarded on, since its audio is pushed as well. If you need both, create a second `PeerConnectionFactory`. - -While a factory sends pushed audio it never opens the recording device, so no microphone is used. Playout of received audio is unaffected. +The recording device is opened only while a microphone track is sent. A factory that sends only pushed audio never opens it. Playout of received audio is unaffected. ::: ### Pushing Audio Data diff --git a/docs/guide/client-api-design.md b/docs/guide/client-api-design.md index 12e681c7b..e435c281a 100644 --- a/docs/guide/client-api-design.md +++ b/docs/guide/client-api-design.md @@ -466,7 +466,7 @@ public interface LocalMedia { LocalAudio microphone(AudioOptions options); // echo cancellation, noise suppression, gain control LocalVideo screen(ScreenOptions options); // a screen or a window from devices() CustomVideo customVideo(String label); - CustomAudio customAudio(String label); // AudioMode.CUSTOM or HEADLESS + CustomAudio customAudio(String label); // any AudioMode LocalVideo videoTrack(String label, VideoTrackSource source); // over a source of the core API LocalAudio audioTrack(String label, AudioTrackSource source); // such as a MediaFileSource's } diff --git a/docs/guide/client-api.md b/docs/guide/client-api.md index 879f629aa..df64ba08c 100644 --- a/docs/guide/client-api.md +++ b/docs/guide/client-api.md @@ -72,15 +72,15 @@ Every timeout of the client API, these two as well as the gathering timeout and ### Audio Modes -WebRTC feeds the audio a factory captures into every audio sender of that factory, so captured audio and audio you push cannot be mixed in one runtime. The runtime's `AudioMode` picks one of them up front: +The runtime's `AudioMode` decides whether it may open the microphone and where it plays remote audio. Every mode sends audio pushed through `CustomAudio`; each sender gets the audio of its own track, so a `DEVICES` runtime can send the microphone and pushed audio side by side, also on one peer: | Mode | Sends | Plays remote audio | Use it for | |---|---|---|---| -| `DEVICES` | the microphone | to the speakers | calls on a desktop | -| `CUSTOM` | audio pushed through `CustomAudio` | to the speakers | applications that generate or relay audio | +| `DEVICES` | the microphone and audio pushed through `CustomAudio` | to the speakers | calls on a desktop | +| `CUSTOM` | audio pushed through `CustomAudio`; the microphone is never opened | to the speakers | applications that generate or relay audio | | `HEADLESS` | audio pushed through `CustomAudio` | to `RemoteAudio.onAudio` handlers only | servers, bots and tests, without any audio hardware | -Asking a runtime for the other kind fails at once with a `ConfigurationException` that names the mode, rather than when the first track is added. Two runtimes can run side by side if an application needs both kinds. +Asking a `CUSTOM` or `HEADLESS` runtime for the microphone fails at once with a `ConfigurationException` that names the mode, rather than when the first track is added. The microphone is opened only while one of its tracks is sent. ## Peers and Signaling @@ -475,7 +475,7 @@ Sender audioSender = peer.addTrack(microphone); | `microphone()`, `microphone(AudioOptions)` | The microphone, with echo cancellation, noise suppression and gain control unless `AudioOptions` say otherwise. `DEVICES` mode only. | | `screen(ScreenOptions)` | A screen or a window, as the [Desktop Video](/guide/video/desktop-capture) guide describes. | | `customVideo(String)` | Frames you push. | -| `customAudio(String)` | Audio you push. `CUSTOM` and `HEADLESS` modes only. | +| `customAudio(String)` | Audio you push, in every mode. | | `videoTrack(String, VideoTrackSource)`, `audioTrack(String, AudioTrackSource)` | A source of the core API, such as one the [media module](/guide/media/media-files) feeds. The source stays yours. | ```java diff --git a/docs/guide/media/media-files.md b/docs/guide/media/media-files.md index dfea4e7cb..8c3967fb5 100644 --- a/docs/guide/media/media-files.md +++ b/docs/guide/media/media-files.md @@ -87,14 +87,7 @@ if (source.getVideoSource() != null) { } ``` -::: warning A factory sends either pushed audio or captured audio -A `PeerConnectionFactory` fed from this module is sending pushed audio and cannot also send audio captured by its `AudioDeviceModule`. Give such a factory a dummy audio layer, and use a second factory if an application needs both: - -```java -AudioDeviceModule audioModule = new AudioDeviceModule(AudioLayer.kDummyAudio); -PeerConnectionFactory factory = new PeerConnectionFactory(audioModule); -``` -::: +The audio of this module is pushed into its track, like that of a `CustomAudioSource`, so the same factory and peer connection can send the microphone beside it. A factory that sends only pushed audio never opens the recording device. ## Reading What a Source Contains diff --git a/webrtc-jni/src/main/cpp/include/JNI_PeerConnectionFactory.h b/webrtc-jni/src/main/cpp/include/JNI_PeerConnectionFactory.h index 8995ce152..de1f3a4b9 100644 --- a/webrtc-jni/src/main/cpp/include/JNI_PeerConnectionFactory.h +++ b/webrtc-jni/src/main/cpp/include/JNI_PeerConnectionFactory.h @@ -39,22 +39,6 @@ extern "C" { JNIEXPORT void JNICALL Java_dev_onvoid_webrtc_PeerConnectionFactory_removeVoiceActivityListenerInternal (JNIEnv *, jobject, jlong); - /* - * Class: dev_onvoid_webrtc_PeerConnectionFactory - * Method: setDeviceCaptureEnabled - * Signature: (Z)V - */ - JNIEXPORT void JNICALL Java_dev_onvoid_webrtc_PeerConnectionFactory_setDeviceCaptureEnabled - (JNIEnv *, jobject, jboolean); - - /* - * Class: dev_onvoid_webrtc_PeerConnectionFactory - * Method: isSinkFedAudioTrack - * Signature: (Ldev/onvoid/webrtc/media/MediaStreamTrack;)Z - */ - JNIEXPORT jboolean JNICALL Java_dev_onvoid_webrtc_PeerConnectionFactory_isSinkFedAudioTrack - (JNIEnv *, jobject, jobject); - /* * Class: dev_onvoid_webrtc_PeerConnectionFactory * Method: createVideoTrack diff --git a/webrtc-jni/src/main/cpp/include/JNI_ToneAudioDeviceModule.h b/webrtc-jni/src/main/cpp/include/JNI_ToneAudioDeviceModule.h index 5fde6224d..747c386f6 100644 --- a/webrtc-jni/src/main/cpp/include/JNI_ToneAudioDeviceModule.h +++ b/webrtc-jni/src/main/cpp/include/JNI_ToneAudioDeviceModule.h @@ -15,6 +15,14 @@ extern "C" { JNIEXPORT void JNICALL Java_dev_onvoid_webrtc_media_audio_ToneAudioDeviceModule_initialize (JNIEnv *, jobject, jint); + /* + * Class: dev_onvoid_webrtc_media_audio_ToneAudioDeviceModule + * Method: isRecording + * Signature: ()Z + */ + JNIEXPORT jboolean JNICALL Java_dev_onvoid_webrtc_media_audio_ToneAudioDeviceModule_isRecording + (JNIEnv *, jobject); + #ifdef __cplusplus } #endif diff --git a/webrtc-jni/src/main/cpp/include/api/ProxyAudioDeviceModule.h b/webrtc-jni/src/main/cpp/include/api/ProxyAudioDeviceModule.h index e1ca067fa..452426f8a 100644 --- a/webrtc-jni/src/main/cpp/include/api/ProxyAudioDeviceModule.h +++ b/webrtc-jni/src/main/cpp/include/api/ProxyAudioDeviceModule.h @@ -33,7 +33,8 @@ namespace jni { // Wraps the AudioDeviceModule a PeerConnectionFactory is created with and - // forwards every call to it. Its purpose is to gate the device capture path. + // forwards every call to it, except for the recorded audio and the control + // of the recording. // // WebRTC's AudioState starts the module's recording as soon as any audio // send stream starts and fans every recorded frame out to every send stream @@ -45,25 +46,21 @@ namespace jni // AudioSendStream::SendAudioData and aborts the process, and interleaves // microphone frames with the pushed audio whenever it does not. // - // Recorded audio therefore never reaches WebRTC's own transport. While - // capture is enabled the proxy hands it to its DeviceAudioCapture, which - // runs it through the factory's AudioProcessing and on to the - // DeviceAudioSources, the sources PeerConnectionFactory.createAudioSource() - // creates. Their tracks' senders receive it through the tracks' sinks, like - // the senders of every other local source. Render calls still go to - // WebRTC, which feeds the same AudioProcessing the playout signal. + // Recorded audio therefore never reaches WebRTC's own transport. The proxy + // hands it to its DeviceAudioCapture, which runs it through the factory's + // AudioProcessing and on to the DeviceAudioSources, the sources + // PeerConnectionFactory.createAudioSource() creates. Their tracks' senders + // receive it through the tracks' sinks, like the senders of every other + // local source, so a factory may send device-captured, pushed and + // forwarded audio side by side. Render calls still go to WebRTC, which + // feeds the same AudioProcessing the playout signal. // - // While capture is disabled, InitRecording() and StartRecording() fail so - // AudioState never opens the device, and recorded data that still arrives - // from the wrapped module (from a recording the application started through - // the Java AudioDeviceModule, which holds the wrapped module directly) is - // dropped before it reaches WebRTC. Playout is never affected. - // - // Capture starts disabled and is enabled only once the application asks for - // device-captured audio, which it can only do through - // PeerConnectionFactory.createAudioSource(). A factory that sends nothing - // but sink-fed audio therefore never opens the recording device, and one - // that sends no audio at all never opens it either. + // WebRTC's InitRecording(), StartRecording() and StopRecording() do not + // reach the device either: they only tell the DeviceAudioCapture whether + // WebRTC asks for a recording. The capture records only while that holds + // and a device source has a sink, so a factory that sends nothing but + // pushed or forwarded audio never opens the recording device. Playout is + // never affected. class ProxyAudioDeviceModule : public webrtc::AudioDeviceModule { public: @@ -73,16 +70,6 @@ namespace jni explicit ProxyAudioDeviceModule(webrtc::scoped_refptr delegate); ~ProxyAudioDeviceModule() override; - // Enables or disables the device capture path. May be called from any - // thread. AudioState starts and stops the module's recording on the - // factory's worker thread, so the switch is made there to keep the two - // from interleaving. The factory creates and owns that thread; this - // proxy learns it when WebRTC registers its transport, which it does - // on the worker thread. Until then no send stream can exist, and the - // switch is made on the calling thread. - void SetCaptureEnabled(bool enabled); - bool CaptureEnabled() const; - // Sets the AudioProcessing the factory is built with, which the // recorded audio runs through. Called before the factory exists. void SetAudioProcessing(webrtc::scoped_refptr processing); @@ -170,16 +157,13 @@ namespace jni std::optional GetStats() const override; private: - void ApplyCaptureEnabled(bool enabled); - // The AudioTransport registered with the wrapped module in place of // the one WebRTC registers with this proxy. Render calls are always - // forwarded to WebRTC; recorded data goes to the DeviceAudioCapture, - // and only while capture is enabled. + // forwarded to WebRTC; recorded data goes to the DeviceAudioCapture. class TransportGate : public webrtc::AudioTransport { public: - TransportGate(const std::atomic & captureEnabled, DeviceAudioCapture & capture); + explicit TransportGate(DeviceAudioCapture & capture); ~TransportGate() override = default; void SetTarget(webrtc::AudioTransport * target); @@ -223,19 +207,15 @@ namespace jni int64_t * ntp_time_ms) override; private: - const std::atomic & captureEnabled_; DeviceAudioCapture & capture_; std::atomic target_; }; webrtc::scoped_refptr delegate_; - std::atomic captureEnabled_; // Declared before the gate, which refers to it. const std::shared_ptr deviceCapture_; TransportGate gate_; bool gateRegistered_; - // The factory's worker thread, set while WebRTC's transport is registered. - std::atomic workerThread_; }; } diff --git a/webrtc-jni/src/main/cpp/include/media/audio/DeviceAudioSource.h b/webrtc-jni/src/main/cpp/include/media/audio/DeviceAudioSource.h index 3d13b1fc5..b827bf73c 100644 --- a/webrtc-jni/src/main/cpp/include/media/audio/DeviceAudioSource.h +++ b/webrtc-jni/src/main/cpp/include/media/audio/DeviceAudioSource.h @@ -21,12 +21,15 @@ #include "api/audio/audio_processing.h" #include "api/audio_options.h" #include "api/media_stream_interface.h" +#include "api/audio/audio_device.h" #include "api/notifier.h" #include "api/scoped_refptr.h" #include "common_audio/resampler/include/push_resampler.h" #include "rtc_base/synchronization/mutex.h" +#include "rtc_base/thread.h" #include "rtc_base/thread_annotations.h" +#include #include #include #include @@ -48,13 +51,22 @@ namespace jni // side is left to WebRTC, which feeds the same AudioProcessing the render // signal, so echo cancellation sees the audio it has to remove. // + // It also decides when the device records: while a source has a sink and + // WebRTC asks for a recording, which it does while any send stream of the + // factory runs and the application has not switched recording off. A + // factory that sends only pushed or forwarded audio therefore never opens + // the recording device. The device is started and stopped on the factory's + // worker thread, where WebRTC asks for it; a change of the sinks posts the + // update there rather than waiting for it, so that no lock is held while + // the device's recording thread is joined. + // // Shared by the factory's ProxyAudioDeviceModule, which feeds it on the // recording thread, and by every DeviceAudioSource it created, which may // outlive the module. - class DeviceAudioCapture + class DeviceAudioCapture : public std::enable_shared_from_this { public: - DeviceAudioCapture() = default; + explicit DeviceAudioCapture(webrtc::scoped_refptr device); DeviceAudioCapture(const DeviceAudioCapture &) = delete; DeviceAudioCapture & operator=(const DeviceAudioCapture &) = delete; @@ -62,8 +74,23 @@ namespace jni // before the factory exists. void SetAudioProcessing(webrtc::scoped_refptr processing); - // Releases the AudioProcessing and every source, once the module is - // gone and no more audio can arrive. + // Sets the factory's worker thread, while WebRTC's transport is + // registered with the device, and clears it when the transport is + // unregistered, before the thread is stopped. Called on that thread. + void SetWorkerThread(webrtc::Thread * workerThread); + + // Records whether WebRTC asks the device to record. Called on the + // worker thread. + void SetRecordingRequested(bool requested); + + // Whether WebRTC asks the device to record, which is what WebRTC + // is told when it asks whether the device records, whatever the + // sources: it decides from that when to ask again. + bool RecordingRequested() const; + + // Stops a recording this object started, and releases the + // AudioProcessing, the device and every source, once the module is + // gone, its worker thread with it, and no more audio can arrive. void Detach(); // Called on the recording thread with 10 ms of interleaved 16-bit @@ -74,10 +101,31 @@ namespace jni // A source is registered while it has at least one sink, and held // for that long, so it is alive whenever audio is handed to it. + // Neither call starts or stops the device: once its lock is + // released, the source calls UpdateRecording(). void AddSource(webrtc::scoped_refptr source); webrtc::scoped_refptr RemoveSource(DeviceAudioSource * source); + // Brings the device's recording in line with the sources and with + // WebRTC's request: at once on the worker thread, else posted to it. + // Must not be called while a source's lock is held. + void UpdateRecording(); + private: + // Starts or stops the device. Runs on the worker thread. + void ApplyRecording(); + + bool HasSources(); + + // Guards the device and the worker thread. + webrtc::Mutex controlMutex_; + webrtc::scoped_refptr device_ RTC_GUARDED_BY(controlMutex_); + webrtc::Thread * workerThread_ RTC_GUARDED_BY(controlMutex_) = nullptr; + + // Set on the worker thread; read by Detach() once it is gone. + std::atomic recordingRequested_ { false }; + std::atomic recordingStarted_ { false }; + // Guards the processing state, which only the recording thread // uses; it changes threads when the device is restarted. webrtc::Mutex processMutex_; @@ -115,6 +163,10 @@ namespace jni void Deliver(const webrtc::AudioFrame & frame, std::optional absoluteCaptureTimeMs); private: + // Adds the sink and registers this source with the capture if it + // is the first one, under the lock. + void AddSinkLocked(webrtc::AudioTrackSinkInterface * sink, bool & registered); + const webrtc::AudioOptions options_; const std::shared_ptr capture_; diff --git a/webrtc-jni/src/main/cpp/src/JNI_PeerConnectionFactory.cpp b/webrtc-jni/src/main/cpp/src/JNI_PeerConnectionFactory.cpp index 2c5d0d9ed..9b10c3bc9 100644 --- a/webrtc-jni/src/main/cpp/src/JNI_PeerConnectionFactory.cpp +++ b/webrtc-jni/src/main/cpp/src/JNI_PeerConnectionFactory.cpp @@ -46,7 +46,6 @@ #include "api/rtc_event_log/rtc_event_log_factory.h" #include "api/ProxyAudioDeviceModule.h" -#include "media/audio/CustomAudioSource.h" #include "media/audio/DeviceAudioSource.h" #include "media/audio/codec/AudioCodecFactories.h" #include "media/video/codec/DefaultVideoCodecFactories.h" @@ -147,10 +146,9 @@ JNIEXPORT void JNICALL Java_dev_onvoid_webrtc_PeerConnectionFactory_initialize } } - // Hand WebRTC a proxy of the module so the device capture path can be - // switched off once this factory sends audio from sink-fed sources. See - // ProxyAudioDeviceModule for why both paths must never feed the same - // send stream. + // Hand WebRTC a proxy of the module, which takes the recorded audio to + // the device sources' tracks instead of to every send stream of the + // factory. See ProxyAudioDeviceModule. auto proxy = jni::ProxyAudioDeviceModule::Create(adm); // No worker thread is supplied: WebRTC is removing support for one that @@ -336,26 +334,6 @@ JNIEXPORT void JNICALL Java_dev_onvoid_webrtc_PeerConnectionFactory_removeVoiceA } } -JNIEXPORT void JNICALL Java_dev_onvoid_webrtc_PeerConnectionFactory_setDeviceCaptureEnabled -(JNIEnv * env, jobject caller, jboolean enabled) -{ - jni::CallFromJava(env, [&]() { - // Keeps the module, which a dispose releases with the factory. - jni::NativeLease factory(env, caller); - CHECK_HANDLE(factory); - - jni::ProxyAudioDeviceModule * audioModule = GetHandle(env, caller, "audioModuleHandle"); - CHECK_HANDLE(audioModule); - - try { - audioModule->SetCaptureEnabled(enabled == JNI_TRUE); - } - catch (...) { - ThrowCxxJavaException(env); - } - }); -} - JNIEXPORT jobject JNICALL Java_dev_onvoid_webrtc_PeerConnectionFactory_createAudioSourceInternal (JNIEnv * env, jobject caller, jobject jAudioOptions) { @@ -418,36 +396,6 @@ JNIEXPORT jobject JNICALL Java_dev_onvoid_webrtc_PeerConnectionFactory_createAud } } -JNIEXPORT jboolean JNICALL Java_dev_onvoid_webrtc_PeerConnectionFactory_isSinkFedAudioTrack -(JNIEnv * env, jobject caller, jobject jTrack) -{ - return jni::CallFromJava(env, [&]() -> jboolean { - if (jTrack == nullptr) { - return JNI_FALSE; - } - - jni::NativeLease track(env, jTrack); - - if (track == nullptr || track->kind() != webrtc::MediaStreamTrackInterface::kAudioKind) { - return JNI_FALSE; - } - - webrtc::AudioSourceInterface * source = static_cast(track.get())->GetSource(); - - if (source == nullptr) { - return JNI_FALSE; - } - - // A remote source hands the track's sinks the audio it decodes, and a - // CustomAudioSource hands them the audio the application pushes. Either way - // a sender of that track is fed through the track rather than by the audio - // device module. - bool sinkFed = source->remote() || dynamic_cast(source) != nullptr; - - return sinkFed ? JNI_TRUE : JNI_FALSE; - }); -} - JNIEXPORT jobject JNICALL Java_dev_onvoid_webrtc_PeerConnectionFactory_createVideoTrack (JNIEnv * env, jobject caller, jstring jlabel, jobject jsource) { diff --git a/webrtc-jni/src/main/cpp/src/JNI_ToneAudioDeviceModule.cpp b/webrtc-jni/src/main/cpp/src/JNI_ToneAudioDeviceModule.cpp index 9a08ec499..9ef31ce35 100644 --- a/webrtc-jni/src/main/cpp/src/JNI_ToneAudioDeviceModule.cpp +++ b/webrtc-jni/src/main/cpp/src/JNI_ToneAudioDeviceModule.cpp @@ -37,3 +37,18 @@ JNIEXPORT void JNICALL Java_dev_onvoid_webrtc_media_audio_ToneAudioDeviceModule_ SetHandle(env, caller, audioModule.release()); }); } + +JNIEXPORT jboolean JNICALL Java_dev_onvoid_webrtc_media_audio_ToneAudioDeviceModule_isRecording +(JNIEnv * env, jobject caller) +{ + jboolean recording = JNI_FALSE; + + jni::CallFromJava(env, [&]() { + jni::NativeLease audioModule(env, caller); + CHECK_HANDLE(audioModule); + + recording = audioModule->Recording() ? JNI_TRUE : JNI_FALSE; + }); + + return recording; +} diff --git a/webrtc-jni/src/main/cpp/src/api/ProxyAudioDeviceModule.cpp b/webrtc-jni/src/main/cpp/src/api/ProxyAudioDeviceModule.cpp index 364f63288..01c3d3c91 100644 --- a/webrtc-jni/src/main/cpp/src/api/ProxyAudioDeviceModule.cpp +++ b/webrtc-jni/src/main/cpp/src/api/ProxyAudioDeviceModule.cpp @@ -32,16 +32,19 @@ namespace jni ProxyAudioDeviceModule::ProxyAudioDeviceModule(webrtc::scoped_refptr delegate) : delegate_(std::move(delegate)), - captureEnabled_(false), - deviceCapture_(std::make_shared()), - gate_(captureEnabled_, *deviceCapture_), - gateRegistered_(false), - workerThread_(nullptr) + deviceCapture_(std::make_shared(delegate_)), + gate_(*deviceCapture_), + gateRegistered_(false) { } ProxyAudioDeviceModule::~ProxyAudioDeviceModule() { + // Stops a recording the capture started, which the module would keep + // running, and with it its callback, and lets the sources live on + // without audio. + deviceCapture_->Detach(); + // WebRTC unregisters its transport before releasing the factory. Should // the wrapped module still point at the gate, detach it so the module // does not keep a dangling pointer once this proxy is gone. A module @@ -51,44 +54,6 @@ namespace jni RTC_LOG(LS_WARNING) << "ProxyAudioDeviceModule: the AudioDeviceModule kept its audio callback. " "Stop its playout and recording before disposing the PeerConnectionFactory."; } - - // No more audio is recorded for the sources, which may live on. - deviceCapture_->Detach(); - } - - void ProxyAudioDeviceModule::SetCaptureEnabled(bool enabled) - { - webrtc::Thread * workerThread = workerThread_.load(std::memory_order_acquire); - - if (workerThread) { - workerThread->BlockingCall([this, enabled]() { - ApplyCaptureEnabled(enabled); - }); - } - else { - ApplyCaptureEnabled(enabled); - } - } - - void ProxyAudioDeviceModule::ApplyCaptureEnabled(bool enabled) - { - captureEnabled_.store(enabled, std::memory_order_release); - - RTC_LOG(LS_INFO) << "ProxyAudioDeviceModule: device capture " << (enabled ? "enabled" : "disabled"); - - if (!enabled && delegate_->Recording()) { - // A recording that is already running was started by the application - // through the Java AudioDeviceModule, which holds the wrapped module - // and does not pass through this proxy. Leave it running; its frames - // are dropped instead. WebRTC cannot have started it, because it only - // starts a recording while capture is enabled. - RTC_LOG(LS_INFO) << "ProxyAudioDeviceModule: dropping recorded audio, send streams are sink-fed"; - } - } - - bool ProxyAudioDeviceModule::CaptureEnabled() const - { - return captureEnabled_.load(std::memory_order_acquire); } void ProxyAudioDeviceModule::SetAudioProcessing(webrtc::scoped_refptr processing) @@ -108,16 +73,26 @@ namespace jni int32_t ProxyAudioDeviceModule::RegisterAudioCallback(webrtc::AudioTransport * audioCallback) { + // WebRtcVoiceEngine registers and unregisters its transport on the + // factory's worker thread, which lives exactly as long as the + // transport is registered. + if (audioCallback == nullptr) { + // The factory goes away. Stop a recording the capture started while + // the worker thread still runs; the module rejects the transport's + // removal while it records. + deviceCapture_->SetRecordingRequested(false); + deviceCapture_->SetWorkerThread(nullptr); + } + gate_.SetTarget(audioCallback); int32_t result = delegate_->RegisterAudioCallback(audioCallback ? &gate_ : nullptr); if (result == 0) { gateRegistered_ = audioCallback != nullptr; - - // WebRtcVoiceEngine registers and unregisters its transport on the - // factory's worker thread. - workerThread_.store(audioCallback ? webrtc::Thread::Current() : nullptr, std::memory_order_release); + } + if (audioCallback != nullptr) { + deviceCapture_->SetWorkerThread(webrtc::Thread::Current()); } return result; @@ -199,24 +174,13 @@ namespace jni int32_t ProxyAudioDeviceModule::RecordingIsAvailable(bool * available) { - if (!CaptureEnabled()) { - if (!available) { - return -1; - } - *available = false; - return 0; - } - return delegate_->RecordingIsAvailable(available); } int32_t ProxyAudioDeviceModule::InitRecording() { - if (!CaptureEnabled()) { - return -1; - } - - return delegate_->InitRecording(); + // The capture initializes the recording once it starts it. + return 0; } bool ProxyAudioDeviceModule::RecordingIsInitialized() const @@ -241,21 +205,28 @@ namespace jni int32_t ProxyAudioDeviceModule::StartRecording() { - if (!CaptureEnabled()) { - return -1; - } + // WebRTC asks for a recording whenever a send stream starts, also one + // fed by a pushed or forwarded track. The device records only while a + // device source has a sink as well. + deviceCapture_->SetRecordingRequested(true); - return delegate_->StartRecording(); + return 0; } int32_t ProxyAudioDeviceModule::StopRecording() { - return delegate_->StopRecording(); + // The last send stream stopped, or the application switched recording + // off through RTCPeerConnection.setAudioRecording(false). + deviceCapture_->SetRecordingRequested(false); + + return 0; } bool ProxyAudioDeviceModule::Recording() const { - return delegate_->Recording(); + // What WebRTC asked for, not what the device does: AudioState asks for + // a recording only while it believes none runs. + return deviceCapture_->RecordingRequested(); } int32_t ProxyAudioDeviceModule::InitSpeaker() @@ -435,9 +406,7 @@ namespace jni // TransportGate - ProxyAudioDeviceModule::TransportGate::TransportGate(const std::atomic & captureEnabled, - DeviceAudioCapture & capture) : - captureEnabled_(captureEnabled), + ProxyAudioDeviceModule::TransportGate::TransportGate(DeviceAudioCapture & capture) : capture_(capture), target_(nullptr) { @@ -478,12 +447,6 @@ namespace jni { newMicLevel = currentMicLevel; - if (!captureEnabled_.load(std::memory_order_acquire)) { - // Drop the frame. Send streams of this factory are fed by their - // tracks' sources, not by the device. - return 0; - } - // Never handed to WebRTC's transport, which would feed it into every // send stream of the factory. The device sources' tracks take it to // their senders instead. diff --git a/webrtc-jni/src/main/cpp/src/media/audio/DeviceAudioSource.cpp b/webrtc-jni/src/main/cpp/src/media/audio/DeviceAudioSource.cpp index 908895e8f..3b377f0fa 100644 --- a/webrtc-jni/src/main/cpp/src/media/audio/DeviceAudioSource.cpp +++ b/webrtc-jni/src/main/cpp/src/media/audio/DeviceAudioSource.cpp @@ -55,6 +55,11 @@ namespace jni // DeviceAudioCapture + DeviceAudioCapture::DeviceAudioCapture(webrtc::scoped_refptr device) : + device_(std::move(device)) + { + } + void DeviceAudioCapture::SetAudioProcessing(webrtc::scoped_refptr processing) { webrtc::MutexLock lock(&processMutex_); @@ -62,9 +67,125 @@ namespace jni processing_ = std::move(processing); } + void DeviceAudioCapture::SetWorkerThread(webrtc::Thread * workerThread) + { + webrtc::MutexLock lock(&controlMutex_); + + workerThread_ = workerThread; + } + + void DeviceAudioCapture::SetRecordingRequested(bool requested) + { + recordingRequested_.store(requested); + + ApplyRecording(); + } + + bool DeviceAudioCapture::RecordingRequested() const + { + return recordingRequested_.load(); + } + + void DeviceAudioCapture::UpdateRecording() + { + { + webrtc::MutexLock lock(&controlMutex_); + + if (workerThread_ == nullptr) { + // No factory runs on the device: either none was created yet, + // so no source can have a sink, or it is going away, and + // Detach() stops what is left. + return; + } + if (!workerThread_->IsCurrent()) { + // Posted while the lock is held, so the thread cannot be + // cleared and stopped in between. The task holds only a weak + // reference: the sources may let this object go meanwhile. + workerThread_->PostTask([weak = weak_from_this()]() { + if (auto self = weak.lock()) { + self->ApplyRecording(); + } + }); + return; + } + } + + ApplyRecording(); + } + + void DeviceAudioCapture::ApplyRecording() + { + webrtc::scoped_refptr device; + + { + webrtc::MutexLock lock(&controlMutex_); + + device = device_; + } + + if (!device) { + return; + } + + const bool record = recordingRequested_.load() && HasSources(); + bool started = recordingStarted_.load(); + + if (started && !device->Recording()) { + // The application stopped the recording through its Java + // AudioDeviceModule; start it again if it is still needed. + recordingStarted_.store(false); + started = false; + } + + if (record && !started) { + if (device->Recording()) { + // The application started the recording through its Java + // AudioDeviceModule. Its audio reaches the sources already, and + // the recording stays the application's to stop. + return; + } + if (device->InitRecording() != 0 || device->StartRecording() != 0) { + RTC_LOG(LS_ERROR) << "DeviceAudioCapture: starting the recording failed"; + return; + } + + recordingStarted_.store(true); + + RTC_LOG(LS_INFO) << "DeviceAudioCapture: recording started"; + } + else if (!record && started) { + device->StopRecording(); + + recordingStarted_.store(false); + + RTC_LOG(LS_INFO) << "DeviceAudioCapture: recording stopped"; + } + } + + bool DeviceAudioCapture::HasSources() + { + webrtc::MutexLock lock(&sourcesMutex_); + + return !sources_.empty(); + } + void DeviceAudioCapture::Detach() { std::vector> sources; + webrtc::scoped_refptr device; + + { + webrtc::MutexLock lock(&controlMutex_); + + workerThread_ = nullptr; + device = std::move(device_); + } + + // The worker thread is gone, so nothing starts the device anymore. A + // recording this object started would otherwise outlive the factory. + if (device && recordingStarted_.exchange(false)) { + device->StopRecording(); + } { webrtc::MutexLock lock(&processMutex_); @@ -192,6 +313,19 @@ namespace jni } void DeviceAudioSource::AddSink(webrtc::AudioTrackSinkInterface * sink) + { + bool registered = false; + + AddSinkLocked(sink, registered); + + // Outside the lock: stopping the device joins its recording thread, + // which may be waiting for this lock to deliver audio. + if (registered) { + capture_->UpdateRecording(); + } + } + + void DeviceAudioSource::AddSinkLocked(webrtc::AudioTrackSinkInterface * sink, bool & registered) { webrtc::MutexLock lock(&mutex_); @@ -206,6 +340,7 @@ namespace jni // this lock while holding its own. if (sinks_.size() == 1) { capture_->AddSource(webrtc::scoped_refptr(this)); + registered = true; } } @@ -231,6 +366,11 @@ namespace jni released = capture_->RemoveSource(this); } } + + // Outside the lock, see AddSink(). + if (released) { + capture_->UpdateRecording(); + } } void DeviceAudioSource::Deliver(const webrtc::AudioFrame & frame, std::optional absoluteCaptureTimeMs) diff --git a/webrtc-media/README.md b/webrtc-media/README.md index 855d1fb3e..b8f9379f8 100644 --- a/webrtc-media/README.md +++ b/webrtc-media/README.md @@ -37,8 +37,8 @@ decoded pictures to a `CustomVideoSource` and 10 ms chunks to a `CustomAudioSour per frame. Frames are paced in real time and carry their own presentation times, so playback keeps the timing of the file rather than the timing of a thread. -A factory that sends media from this module sends pushed audio, so it cannot also send audio -captured by its `AudioDeviceModule`. Use a separate factory if you need both. +The audio of this module is pushed into its track, so the same factory can send the microphone +beside it. ## FFmpeg diff --git a/webrtc-media/src/main/java/dev/onvoid/webrtc/media/player/MediaFileSource.java b/webrtc-media/src/main/java/dev/onvoid/webrtc/media/player/MediaFileSource.java index 42305cc8a..a26629925 100644 --- a/webrtc-media/src/main/java/dev/onvoid/webrtc/media/player/MediaFileSource.java +++ b/webrtc-media/src/main/java/dev/onvoid/webrtc/media/player/MediaFileSource.java @@ -49,9 +49,8 @@ * {@link #getVideoSource()} and {@link #getAudioSource()} for {@code null} * before making a track, or ask {@link #getInfo()} first. *

- * A factory fed from here is sending pushed audio, so it cannot also send - * audio captured by its {@code AudioDeviceModule}. Use a separate factory if - * an application needs both. + * The audio is pushed into its track, so the same factory may send the + * microphone beside it. *

* Closing this releases the player and both media sources. Dispose of any * tracks made from them first. diff --git a/webrtc-media/src/main/java/dev/onvoid/webrtc/media/player/MediaPlayer.java b/webrtc-media/src/main/java/dev/onvoid/webrtc/media/player/MediaPlayer.java index 449a54dfc..08ff1ca80 100644 --- a/webrtc-media/src/main/java/dev/onvoid/webrtc/media/player/MediaPlayer.java +++ b/webrtc-media/src/main/java/dev/onvoid/webrtc/media/player/MediaPlayer.java @@ -53,9 +53,8 @@ * player.play(); * } *

- * A factory fed by this player is sending pushed audio, so it cannot also send - * audio captured by its {@code AudioDeviceModule}. Use a separate factory if - * an application needs both. + * The audio is pushed into its track, so the same factory may send the + * microphone beside it. * * @author Alex Andres */ diff --git a/webrtc/src/main/java/dev/onvoid/webrtc/PeerConnectionFactory.java b/webrtc/src/main/java/dev/onvoid/webrtc/PeerConnectionFactory.java index 18e9ecb81..3b0e6e29c 100644 --- a/webrtc/src/main/java/dev/onvoid/webrtc/PeerConnectionFactory.java +++ b/webrtc/src/main/java/dev/onvoid/webrtc/PeerConnectionFactory.java @@ -50,21 +50,18 @@ * It provides factory methods for {@link RTCPeerConnection} and audio/video * {@link MediaStreamTrack}s. *

- * A factory sends audio from exactly one kind of input. Either its - * {@link AudioDeviceModuleBase audio device module} captures the audio, which - * is what tracks created from {@link #createAudioSource(AudioOptions)} send, - * or the audio is pushed into the senders, which is what a - * {@link CustomAudioSource} and a track forwarded from a remote peer do. - * WebRTC feeds device-captured audio into every audio sender of a factory, so - * a sender that is fed both ways hits a race inside WebRTC that aborts the - * process. The factory therefore commits to the kind that is used first and - * rejects the other with an {@link IllegalStateException}. Applications that - * need both create a second factory. + * A factory may send audio of every kind side by side, also on one peer + * connection: audio its {@link AudioDeviceModuleBase audio device module} + * captures, which is what tracks created from + * {@link #createAudioSource(AudioOptions)} send, audio pushed through a + * {@link CustomAudioSource}, and tracks forwarded from a remote peer. Each + * sender gets the audio of its own track only. *

- * The recording device is opened only once - * {@link #createAudioSource(AudioOptions)} is called, since that is the only - * way to send device-captured audio. A factory that sends pushed audio, or no - * audio at all, never opens it. Playout of received audio is never affected. + * The recording device is opened only while a track of a source created by + * {@link #createAudioSource(AudioOptions)} is sent, or has a sink, and some + * audio sender of the factory is sending. A factory that sends only pushed or + * forwarded audio, or no audio at all, never opens it. Playout of received + * audio is never affected. *

* The constructors cover the common combinations of modules. A {@link * #builder() builder} sets any of them, including the video codecs. @@ -82,19 +79,6 @@ public class PeerConnectionFactory extends DisposableNativeObject { } } - /** - * The kind of input that provides the audio this factory sends. - */ - private enum AudioInputMode { - - /** The audio device module captures the audio. */ - DEVICE, - - /** The application pushes the audio through a {@link CustomAudioSource}. */ - CUSTOM - - } - @SuppressWarnings("unused") private long networkThreadHandle; @@ -114,12 +98,6 @@ private enum AudioInputMode { */ private final Map voiceActivityListeners = new IdentityHashMap<>(); - /** Guards {@link #audioInputMode}. */ - private final Object audioInputLock = new Object(); - - /** Set the first time either kind of audio input is used; never reset. */ - private AudioInputMode audioInputMode; - /** * Creates an instance of PeerConnectionFactory. @@ -260,23 +238,16 @@ public static Builder builder() { * The audio reaches the senders of these tracks through the tracks, after * the factory's audio processing: echo cancellation, noise suppression and * gain control as the options ask for. A sink added to such a track gets - * the same audio, on WebRTC's capture thread, while the device records. - *

- * Calling this commits the factory to device-captured audio and lets it open - * the recording device; see the class description. + * the same audio, on WebRTC's capture thread, while the device records; + * see the class description for when it does. * * @param options Audio options to control the audio processing. * * @return The created audio source. - * - * @throws IllegalStateException If this factory already sends audio from a - * {@link CustomAudioSource}. */ public AudioTrackSource createAudioSource(AudioOptions options) { Objects.requireNonNull(options, "AudioOptions is null"); - requireAudioInputMode(AudioInputMode.DEVICE); - return createAudioSourceInternal(options); } @@ -285,87 +256,19 @@ public AudioTrackSource createAudioSource(AudioOptions options) { * {@link RTCPeerConnection} using the {@link RTCPeerConnection#addTrack * addTrack} or {@link RTCPeerConnection#addTransceiver addTransceiver} * methods. - *

- * Passing a {@link CustomAudioSource} commits the factory to pushed audio; - * see the class description. * * @param label The identifier string of the audio track. * @param source The audio source that provides audio data. * * @return The created audio track. - * - * @throws IllegalStateException If the source is a {@link CustomAudioSource} - * and this factory already sends audio - * captured by its audio device module. */ public AudioTrack createAudioTrack(String label, AudioTrackSource source) { Objects.requireNonNull(label, "Audio track label is null"); Objects.requireNonNull(source, "AudioTrackSource is null"); - if (source instanceof CustomAudioSource) { - requireAudioInputMode(AudioInputMode.CUSTOM); - } - return createAudioTrackInternal(label, source); } - /** - * Commits this factory to the given kind of audio input, or verifies that - * it is already committed to it. Committing to {@link AudioInputMode#CUSTOM} - * switches off capture from the audio device module. - * - * @param mode The kind of audio input about to be used. - * - * @throws IllegalStateException If the factory is committed to the other - * kind. - */ - private void requireAudioInputMode(AudioInputMode mode) { - synchronized (audioInputLock) { - if (audioInputMode == mode) { - return; - } - if (audioInputMode == AudioInputMode.CUSTOM) { - throw new IllegalStateException("This PeerConnectionFactory already sends audio pushed through a " - + "CustomAudioSource. WebRTC feeds the audio captured by a factory's AudioDeviceModule into " - + "every audio sender of that factory, so device-captured and custom audio sources cannot be " - + "combined in one factory. Create a separate PeerConnectionFactory for device-captured audio."); - } - if (audioInputMode == AudioInputMode.DEVICE) { - throw new IllegalStateException("This PeerConnectionFactory already sends audio captured by its " - + "AudioDeviceModule. WebRTC feeds that audio into every audio sender of the factory, so " - + "device-captured and custom audio sources cannot be combined in one factory. Create a " - + "separate PeerConnectionFactory for CustomAudioSource tracks."); - } - - // The recording device stays closed until the application asks for - // device-captured audio, which is what keeps captured frames away - // from senders fed by a custom source or by a forwarded remote track. - setDeviceCaptureEnabled(mode == AudioInputMode.DEVICE); - - audioInputMode = mode; - } - } - - /** - * Commits this factory to pushed audio if the given track is an audio track - * whose audio is pushed into its sender rather than captured by the audio - * device module. Tracks backed by a {@link CustomAudioSource} and tracks - * received from a remote peer and forwarded on are of that kind. Called - * before a track becomes a sender of one of this factory's peer - * connections; any other track is ignored. - * - * @param track The track about to be sent, may be {@code null}. - * - * @throws IllegalStateException If the track is of that kind and this - * factory already sends audio captured by its - * audio device module. - */ - void commitAudioInput(MediaStreamTrack track) { - if (track != null && isSinkFedAudioTrack(track)) { - requireAudioInputMode(AudioInputMode.CUSTOM); - } - } - /** * Creates a new {@link VideoTrack}. The video track can be added to the * {@link RTCPeerConnection} using the {@link RTCPeerConnection#addTrack @@ -394,8 +297,7 @@ public RTCPeerConnection createPeerConnection(RTCConfiguration config, observer); if (peerConnection != null) { - // The connection reports back which kind of audio its senders are - // about to send, so this factory can reject a mix of both kinds. + // A close of the connection keeps the factory's threads. peerConnection.setFactory(this); } @@ -534,27 +436,6 @@ private native void initialize(Map fieldTrials, private native RTCPeerConnection createPeerConnectionInternal( RTCConfiguration config, PeerConnectionObserver observer); - /** - * Returns whether the given track is an audio track whose audio reaches a - * sender through the track's sinks instead of through the audio device - * module. That is the case for a {@link CustomAudioSource} and for a track - * received from a remote peer. - * - * @param track The track to classify, must not be {@code null}. - * - * @return True if the track is audio and its source feeds sinks. - */ - private native boolean isSinkFedAudioTrack(MediaStreamTrack track); - - /** - * Enables or disables the capture path of the audio device module WebRTC - * uses for this factory. While disabled, WebRTC cannot start a recording - * and recorded audio never reaches the factory's audio senders. - * - * @param enabled Whether device-captured audio may reach the audio senders. - */ - private native void setDeviceCaptureEnabled(boolean enabled); - /** * Builds a {@link PeerConnectionFactory}. Each module that is not set is diff --git a/webrtc/src/main/java/dev/onvoid/webrtc/RTCPeerConnection.java b/webrtc/src/main/java/dev/onvoid/webrtc/RTCPeerConnection.java index 7719c87a5..6974ce0c0 100644 --- a/webrtc/src/main/java/dev/onvoid/webrtc/RTCPeerConnection.java +++ b/webrtc/src/main/java/dev/onvoid/webrtc/RTCPeerConnection.java @@ -16,9 +16,6 @@ package dev.onvoid.webrtc; -import static java.util.Objects.isNull; -import static java.util.Objects.nonNull; - import dev.onvoid.webrtc.internal.NativeObject; import dev.onvoid.webrtc.media.MediaStreamTrack; import dev.onvoid.webrtc.media.MediaType; @@ -42,9 +39,11 @@ public class RTCPeerConnection extends NativeObject { private long observerHandle; /** - * The factory that created this connection. Set right after construction; - * null only if a connection was obtained some other way. + * The factory that created this connection, whose threads a close keeps + * while it is handed on. Set right after construction; null only if a + * connection was obtained some other way. */ + @SuppressWarnings("unused") private PeerConnectionFactory factory; @@ -56,8 +55,7 @@ private RTCPeerConnection() { } /** - * Sets the factory that created this connection, so that tracks added to it - * can be reported back to that factory. + * Sets the factory that created this connection. * * @param factory The creating factory. */ @@ -101,19 +99,9 @@ void setFactory(PeerConnectionFactory factory) { * to. * * @return The RTCRtpSender which will be used to transmit the media data. - * - * @throws IllegalStateException If the track is an audio track that pushes - * its audio, for example one received from a - * remote peer, while the creating factory - * already sends audio captured by its audio - * device module. See {@link - * PeerConnectionFactory} for why a factory - * sends only one kind of audio input. */ public RTCRtpSender addTrack(MediaStreamTrack track, List streamIds) { - commitAudioInput(track); - return addTrackInternal(track, streamIds); } @@ -139,26 +127,9 @@ public RTCRtpSender addTrack(MediaStreamTrack track, * * @return The RTCRtpTransceiver which will be used to transmit and receive * the media data. - * - * @throws IllegalStateException If the transceiver sends and the track is an - * audio track that pushes its audio, for - * example one received from a remote peer, - * while the creating factory already sends - * audio captured by its audio device module. - * See {@link PeerConnectionFactory} for why a - * factory sends only one kind of audio input. */ public RTCRtpTransceiver addTransceiver(MediaStreamTrack track, RTCRtpTransceiverInit init) { - // A transceiver that only receives never sends the track's audio. Its - // direction can be changed later through RTCRtpTransceiver.setDirection, - // which is not covered here. - if (isNull(init) || isNull(init.direction) - || init.direction == RTCRtpTransceiverDirection.SEND_RECV - || init.direction == RTCRtpTransceiverDirection.SEND_ONLY) { - commitAudioInput(track); - } - return addTransceiverInternal(track, init); } @@ -179,19 +150,6 @@ public RTCRtpTransceiver addTransceiver(MediaType mediaType, RTCRtpTransceiverIn return addTransceiverOfType(mediaType, init); } - /** - * Reports a track that is about to be sent to the factory that created this - * connection, which rejects a track whose audio is pushed while the factory - * already sends audio captured by its audio device module. - * - * @param track The track about to be sent, may be {@code null}. - */ - private void commitAudioInput(MediaStreamTrack track) { - if (nonNull(factory)) { - factory.commitAudioInput(track); - } - } - /** * Creates a new RTCDataChannel object with the given label. The * RTCDataChannelInit dictionary can be used to configure properties of the diff --git a/webrtc/src/main/java/dev/onvoid/webrtc/client/AudioMode.java b/webrtc/src/main/java/dev/onvoid/webrtc/client/AudioMode.java index 2b3f9205b..62ee81e7b 100644 --- a/webrtc/src/main/java/dev/onvoid/webrtc/client/AudioMode.java +++ b/webrtc/src/main/java/dev/onvoid/webrtc/client/AudioMode.java @@ -17,17 +17,22 @@ package dev.onvoid.webrtc.client; /** - * Where the audio a runtime sends comes from. WebRTC feeds the audio a factory - * captures into every audio sender of that factory, so captured and pushed - * audio cannot be mixed within one runtime, and the choice is made when it is - * built. + * Where the audio a runtime sends can come from, and where it plays the audio + * it receives. The choice is made when the runtime is built. */ public enum AudioMode { - /** Capture the microphone and play to the speakers. */ + /** + * Capture the microphone and play to the speakers. Audio pushed through + * {@link CustomAudio} can be sent as well, beside the microphone, also on + * one peer. + */ DEVICES, - /** Send audio pushed through {@link CustomAudio} and play remote audio to the speakers. */ + /** + * Send audio pushed through {@link CustomAudio} and play remote audio to the + * speakers. The microphone is never opened. + */ CUSTOM, /** diff --git a/webrtc/src/main/java/dev/onvoid/webrtc/client/LocalMedia.java b/webrtc/src/main/java/dev/onvoid/webrtc/client/LocalMedia.java index c2789235e..f6d2b9afe 100644 --- a/webrtc/src/main/java/dev/onvoid/webrtc/client/LocalMedia.java +++ b/webrtc/src/main/java/dev/onvoid/webrtc/client/LocalMedia.java @@ -124,9 +124,8 @@ public interface LocalMedia { * * @param label the track label. * @param source the source that feeds the track. A {@link CustomAudioSource} - * needs a runtime built with {@link AudioMode#CUSTOM} or - * {@link AudioMode#HEADLESS}, any other source {@link - * AudioMode#DEVICES}. + * fits any runtime, any other source needs a runtime built + * with {@link AudioMode#DEVICES}. * * @return the track. * @@ -141,9 +140,6 @@ public interface LocalMedia { * @param label the track label. * * @return the track. - * - * @throws ConfigurationException if the runtime's {@link AudioMode} is - * {@link AudioMode#DEVICES}. */ CustomAudio customAudio(String label); diff --git a/webrtc/src/main/java/dev/onvoid/webrtc/client/LocalMediaImpl.java b/webrtc/src/main/java/dev/onvoid/webrtc/client/LocalMediaImpl.java index fbb20616b..38043e46a 100644 --- a/webrtc/src/main/java/dev/onvoid/webrtc/client/LocalMediaImpl.java +++ b/webrtc/src/main/java/dev/onvoid/webrtc/client/LocalMediaImpl.java @@ -153,8 +153,8 @@ public LocalAudio microphone(AudioOptions options) { try (Creation creation = begin()) { if (rtc.audioMode != AudioMode.DEVICES) { throw new ConfigurationException("The runtime was built with AudioMode." + rtc.audioMode - + ", which sends pushed audio only. Build it with AudioMode.DEVICES to capture the microphone, " - + "or use customAudio() to push audio"); + + ", which never opens the microphone. Build it with AudioMode.DEVICES to capture the " + + "microphone, or use customAudio() to push audio"); } AudioTrackSource source = rtc.factory.createAudioSource(options); @@ -242,10 +242,6 @@ public LocalAudio audioTrack(String label, AudioTrackSource source) { try (Creation creation = begin()) { boolean pushed = source instanceof CustomAudioSource; - if (pushed && rtc.audioMode == AudioMode.DEVICES) { - throw new ConfigurationException("A CustomAudioSource needs a runtime built with AudioMode.CUSTOM or " - + "AudioMode.HEADLESS; this one captures the microphone"); - } if (!pushed && rtc.audioMode != AudioMode.DEVICES) { throw new ConfigurationException("A captured audio source needs a runtime built with AudioMode.DEVICES; " + "this one was built with AudioMode." + rtc.audioMode); @@ -263,12 +259,6 @@ public CustomAudio customAudio(String label) { requireNonNull(label, "label"); try (Creation creation = begin()) { - if (rtc.audioMode == AudioMode.DEVICES) { - throw new ConfigurationException("The runtime was built with AudioMode.DEVICES, which sends the " - + "microphone. WebRTC feeds captured audio into every audio sender of a runtime, so pushed " - + "audio needs a runtime built with AudioMode.CUSTOM or AudioMode.HEADLESS"); - } - CustomAudioSource source = new CustomAudioSource(); try { diff --git a/webrtc/src/test/java/dev/onvoid/webrtc/DeviceAudioSendTests.java b/webrtc/src/test/java/dev/onvoid/webrtc/DeviceAudioSendTests.java index 33886cb7f..3efbee0c7 100644 --- a/webrtc/src/test/java/dev/onvoid/webrtc/DeviceAudioSendTests.java +++ b/webrtc/src/test/java/dev/onvoid/webrtc/DeviceAudioSendTests.java @@ -16,6 +16,7 @@ package dev.onvoid.webrtc; +import static org.junit.jupiter.api.Assertions.assertFalse; import static org.junit.jupiter.api.Assertions.assertNotNull; import static org.junit.jupiter.api.Assertions.assertTrue; @@ -23,6 +24,7 @@ import dev.onvoid.webrtc.media.audio.AudioTrack; import dev.onvoid.webrtc.media.audio.AudioTrackSink; import dev.onvoid.webrtc.media.audio.AudioTrackSource; +import dev.onvoid.webrtc.media.audio.CustomAudioSource; import dev.onvoid.webrtc.media.audio.ToneAudioDeviceModule; import org.junit.jupiter.api.Test; @@ -33,13 +35,16 @@ import java.util.Map; import java.util.concurrent.CountDownLatch; import java.util.concurrent.TimeUnit; +import java.util.concurrent.atomic.AtomicBoolean; import java.util.concurrent.atomic.AtomicReference; +import java.util.function.BooleanSupplier; import java.util.function.Predicate; /** * Audio captured by the factory's audio device module reaches the senders * through the track of a device source, like the audio of every other local - * source. The module here records a tone in place of a microphone. + * source, so a factory sends it beside pushed audio. The module here records a + * tone in place of a microphone. */ class DeviceAudioSendTests extends TestBase { @@ -133,6 +138,116 @@ void echoStatisticsReported() throws Exception { } } + @Test + void deviceAndPushedTracksOnOnePeer() throws Exception { + try (DeviceCall call = new DeviceCall(new AudioOptions(), true, true)) { + call.connect(); + call.startPushing(); + + // The callee's transceivers follow the order of the offer: the + // microphone first, then the pushed silence. + RTCRtpReceiver[] receivers = call.getReceivers(); + AudioTrack microphone = (AudioTrack) receivers[0].getTrack(); + AudioTrack pushed = (AudioTrack) receivers[1].getTrack(); + + CountDownLatch microphoneAudible = new CountDownLatch(5); + CountDownLatch pushedReceived = new CountDownLatch(50); + AtomicBoolean pushedAudible = new AtomicBoolean(); + + AudioTrackSink microphoneSink = (data, bits, rate, channels, frames) -> { + if (isAudible(data, bits, channels, frames)) { + microphoneAudible.countDown(); + } + }; + // Recorded audio fed into the pushed track's sender as well would + // make the silence audible. + AudioTrackSink pushedSink = (data, bits, rate, channels, frames) -> { + if (isAudible(data, bits, channels, frames)) { + pushedAudible.set(true); + } + pushedReceived.countDown(); + }; + + microphone.addSink(microphoneSink); + pushed.addSink(pushedSink); + + try { + assertTrue(microphoneAudible.await(TIMEOUT_SECONDS, TimeUnit.SECONDS), + "No recorded audio at the callee"); + assertTrue(pushedReceived.await(TIMEOUT_SECONDS, TimeUnit.SECONDS), + "No pushed audio at the callee"); + assertFalse(pushedAudible.get(), "Recorded audio reached the pushed track"); + } + finally { + microphone.removeSink(microphoneSink); + pushed.removeSink(pushedSink); + + for (RTCRtpReceiver receiver : receivers) { + receiver.dispose(); + } + } + } + } + + @Test + void pushedAudioAloneNeverRecords() throws Exception { + try (DeviceCall call = new DeviceCall(new AudioOptions(), false, true)) { + call.connect(); + call.startPushing(); + + // A send stream runs, so WebRTC asks for a recording, but no device + // source needs it. + assertNotNull(call.awaitStats(call.customSender, RTCStatsType.OUTBOUND_RTP, + attributes -> number(attributes, "packetsSent") > 0), "Nothing was sent"); + assertFalse(call.audioModule.isRecording(), "The device records for pushed audio"); + } + } + + @Test + void recordingFollowsSetAudioRecording() throws Exception { + try (DeviceCall call = new DeviceCall(new AudioOptions(), true, false)) { + call.connect(); + + assertTrue(await(call.audioModule::isRecording), "The device does not record"); + + call.caller.getPeerConnection().setAudioRecording(false); + assertTrue(await(() -> !call.audioModule.isRecording()), "Recording was not switched off"); + + call.caller.getPeerConnection().setAudioRecording(true); + assertTrue(await(call.audioModule::isRecording), "Recording was not switched on again"); + } + } + + @Test + void recordingStopsWithLastDeviceTrack() throws Exception { + try (DeviceCall call = new DeviceCall(new AudioOptions(), true, true)) { + call.connect(); + call.startPushing(); + + assertTrue(await(call.audioModule::isRecording), "The device does not record"); + + // The pushed track keeps a send stream, and WebRTC's request for a + // recording, alive; the microphone track's sink is gone. + call.caller.getPeerConnection().removeTrack(call.sender); + + assertTrue(await(() -> !call.audioModule.isRecording()), "The device still records"); + } + } + + private static boolean await(BooleanSupplier condition) throws InterruptedException { + long deadline = System.nanoTime() + TimeUnit.SECONDS.toNanos(TIMEOUT_SECONDS); + + while (System.nanoTime() < deadline) { + if (condition.getAsBoolean()) { + return true; + } + + Thread.sleep(20); + } + + return condition.getAsBoolean(); + } + private static boolean isAudible(byte[] data, int bits, int channels, int frames) { if (bits != 16) { return false; @@ -159,8 +274,9 @@ private static double number(Map attributes, String name) { /** - * A call that sends the track of a device source from the caller to the - * callee, both on a factory whose audio device module records a tone. + * A call that sends the track of a device source, a track of pushed + * silence, or both, from the caller to the callee, all on one factory whose + * audio device module records a tone. */ private static class DeviceCall implements AutoCloseable { @@ -178,12 +294,24 @@ private static class DeviceCall implements AutoCloseable { final RTCRtpSender sender; + final CustomAudioSource customSource; + + final AudioTrack customTrack; + + final RTCRtpSender customSender; + + private volatile boolean pushing; + + private Thread pusher; + DeviceCall(AudioOptions options) { + this(options, true, false); + } + + DeviceCall(AudioOptions options, boolean device, boolean pushed) { audioModule = new ToneAudioDeviceModule(440); factory = new PeerConnectionFactory(audioModule); - source = factory.createAudioSource(options); - track = factory.createAudioTrack("microphone", source); caller = new TestPeerConnection(factory); callee = new TestPeerConnection(factory); @@ -191,7 +319,57 @@ private static class DeviceCall implements AutoCloseable { caller.setRemotePeerConnection(callee); callee.setRemotePeerConnection(caller); - sender = caller.getPeerConnection().addTrack(track, Collections.singletonList("stream")); + if (device) { + source = factory.createAudioSource(options); + track = factory.createAudioTrack("microphone", source); + sender = caller.getPeerConnection().addTrack(track, Collections.singletonList("stream")); + } + else { + source = null; + track = null; + sender = null; + } + + if (pushed) { + customSource = new CustomAudioSource(); + customTrack = factory.createAudioTrack("pushed", customSource); + customSender = caller.getPeerConnection().addTrack(customTrack, Collections.singletonList("stream")); + } + else { + customSource = null; + customTrack = null; + customSender = null; + } + } + + /** + * Starts pushing 10 ms of silence every 10 ms into the custom source. + */ + void startPushing() { + pushing = true; + pusher = new Thread(() -> { + byte[] silence = new byte[480 * 2]; + long next = System.nanoTime(); + + while (pushing) { + customSource.pushAudio(silence, 16, 48000, 1, 480); + + next += 10_000_000L; + + long wait = next - System.nanoTime(); + + if (wait > 0) { + try { + TimeUnit.NANOSECONDS.sleep(wait); + } + catch (InterruptedException e) { + return; + } + } + } + }, "DeviceCall-pusher"); + pusher.setDaemon(true); + pusher.start(); } void connect() throws Exception { @@ -203,31 +381,51 @@ void connect() throws Exception { } /** - * Returns the callee's receiver of the track, which is the caller's to - * dispose. + * Returns the callee's receiver of the first track, which is the + * caller's to dispose. */ RTCRtpReceiver getReceiver() { + RTCRtpReceiver[] receivers = getReceivers(); + + for (int i = 1; i < receivers.length; i++) { + receivers[i].dispose(); + } + + return receivers[0]; + } + + /** + * Returns the callee's receivers, in the order of the offer, which are + * the caller's to dispose. + */ + RTCRtpReceiver[] getReceivers() { RTCRtpTransceiver[] transceivers = callee.getPeerConnection().getTransceivers(); - RTCRtpReceiver receiver = transceivers[0].getReceiver(); + RTCRtpReceiver[] receivers = new RTCRtpReceiver[transceivers.length]; - for (RTCRtpTransceiver transceiver : transceivers) { - transceiver.dispose(); + for (int i = 0; i < transceivers.length; i++) { + receivers[i] = transceivers[i].getReceiver(); + transceivers[i].dispose(); } - return receiver; + return receivers; } Map awaitMediaSourceStats(Predicate> condition) throws InterruptedException { + return awaitStats(sender, RTCStatsType.MEDIA_SOURCE, condition); + } + + Map awaitStats(RTCRtpSender statsSender, RTCStatsType type, + Predicate> condition) throws InterruptedException { long deadline = System.nanoTime() + TimeUnit.SECONDS.toNanos(TIMEOUT_SECONDS); while (System.nanoTime() < deadline) { CountDownLatch done = new CountDownLatch(1); AtomicReference> found = new AtomicReference<>(); - caller.getPeerConnection().getStats(sender, report -> { + caller.getPeerConnection().getStats(statsSender, report -> { for (RTCStats stats : report.getStats().values()) { - if (stats.getType() == RTCStatsType.MEDIA_SOURCE && condition.test(stats.getAttributes())) { + if (stats.getType() == type && condition.test(stats.getAttributes())) { found.set(stats.getAttributes()); } } @@ -247,14 +445,32 @@ Map awaitMediaSourceStats(Predicate> conditi } @Override - public void close() { - sender.dispose(); + public void close() throws InterruptedException { + pushing = false; + + if (pusher != null) { + pusher.join(); + } + + if (sender != null) { + sender.dispose(); + } + if (customSender != null) { + customSender.dispose(); + } caller.close(); callee.close(); - track.dispose(); - source.dispose(); + if (track != null) { + track.dispose(); + source.dispose(); + } + if (customTrack != null) { + customTrack.dispose(); + customSource.dispose(); + } + factory.dispose(); audioModule.dispose(); } diff --git a/webrtc/src/test/java/dev/onvoid/webrtc/PeerConnectionFactoryTests.java b/webrtc/src/test/java/dev/onvoid/webrtc/PeerConnectionFactoryTests.java index 6ec9d781b..1563798cf 100644 --- a/webrtc/src/test/java/dev/onvoid/webrtc/PeerConnectionFactoryTests.java +++ b/webrtc/src/test/java/dev/onvoid/webrtc/PeerConnectionFactoryTests.java @@ -190,61 +190,41 @@ void createAudioTrack() { } @Test - void customAudioSourceAfterDeviceAudioSourceIsRejected() { - // WebRTC feeds device-captured audio into every audio sender of a - // factory. A CustomAudioSource track would be fed twice, which aborts - // the process inside WebRTC, so the factory must refuse it up front. + void deviceAndCustomSourcesShareFactory() { + // Each sender gets the audio of its own track, so a factory sends + // device-captured and pushed audio side by side, also on one connection. AudioDeviceModule audioModule = new AudioDeviceModule(AudioLayer.kDummyAudio); PeerConnectionFactory audioFactory = new PeerConnectionFactory(audioModule); CustomAudioSource customSource = new CustomAudioSource(); try { AudioTrackSource deviceSource = audioFactory.createAudioSource(new AudioOptions()); - - IllegalStateException e = assertThrows(IllegalStateException.class, () -> { - audioFactory.createAudioTrack("customTrack", customSource); - }); - assertTrue(e.getMessage().contains("CustomAudioSource"), e.getMessage()); - - // The device path stays usable. AudioTrack deviceTrack = audioFactory.createAudioTrack("deviceTrack", deviceSource); - assertNotNull(deviceTrack); - - deviceTrack.dispose(); - deviceSource.dispose(); - } - finally { - customSource.dispose(); - audioFactory.dispose(); - audioModule.dispose(); - } - } + AudioTrack customTrack = audioFactory.createAudioTrack("customTrack", customSource); - @Test - void deviceAudioSourceAfterCustomAudioSourceIsRejected() { - AudioDeviceModule audioModule = new AudioDeviceModule(AudioLayer.kDummyAudio); - PeerConnectionFactory audioFactory = new PeerConnectionFactory(audioModule); - CustomAudioSource customSource = new CustomAudioSource(); - CustomAudioSource otherCustomSource = new CustomAudioSource(); + // Either order of creation is fine. + AudioTrackSource otherDeviceSource = audioFactory.createAudioSource(new AudioOptions()); - try { - AudioTrack customTrack = audioFactory.createAudioTrack("customTrack", customSource); - assertNotNull(customTrack); + RTCPeerConnection peerConnection = audioFactory.createPeerConnection( + new RTCConfiguration(), candidate -> { }); - IllegalStateException e = assertThrows(IllegalStateException.class, () -> { - audioFactory.createAudioSource(new AudioOptions()); - }); - assertTrue(e.getMessage().contains("AudioDeviceModule"), e.getMessage()); + RTCRtpSender deviceSender = peerConnection.addTrack(deviceTrack, + Collections.singletonList("stream0")); + RTCRtpTransceiver customTransceiver = peerConnection.addTransceiver(customTrack, + new RTCRtpTransceiverInit()); - // Any number of custom sources may share the factory. - AudioTrack otherCustomTrack = audioFactory.createAudioTrack("otherCustomTrack", otherCustomSource); - assertNotNull(otherCustomTrack); + assertNotNull(deviceSender); + assertNotNull(customTransceiver); - otherCustomTrack.dispose(); + customTransceiver.dispose(); + deviceSender.dispose(); + peerConnection.close(); customTrack.dispose(); + deviceTrack.dispose(); + otherDeviceSource.dispose(); + deviceSource.dispose(); } finally { - otherCustomSource.dispose(); customSource.dispose(); audioFactory.dispose(); audioModule.dispose(); @@ -252,10 +232,10 @@ void deviceAudioSourceAfterCustomAudioSourceIsRejected() { } @Test - void sinkFedTrackAddedToDeviceAudioFactoryIsRejected() { - // A track whose audio is pushed rather than captured, here one backed by - // a CustomAudioSource but equally one forwarded from a remote peer, may - // not become a sender of a factory that sends device-captured audio. + void foreignPushedTrackBesideDeviceTrack() { + // A track whose audio is pushed and that another factory created, as a + // track forwarded from a remote peer is, may be sent beside a + // device-captured one. AudioDeviceModule deviceModule = new AudioDeviceModule(AudioLayer.kDummyAudio); AudioDeviceModule customModule = new AudioDeviceModule(AudioLayer.kDummyAudio); PeerConnectionFactory deviceFactory = new PeerConnectionFactory(deviceModule); @@ -263,41 +243,25 @@ void sinkFedTrackAddedToDeviceAudioFactoryIsRejected() { CustomAudioSource customSource = new CustomAudioSource(); try { - AudioTrackSource deviceSource = deviceFactory.createAudioSource(new AudioOptions()); - AudioTrack deviceTrack = deviceFactory.createAudioTrack("deviceTrack", deviceSource); AudioTrack customTrack = customFactory.createAudioTrack("customTrack", customSource); RTCPeerConnection peerConnection = deviceFactory.createPeerConnection( new RTCConfiguration(), candidate -> { }); - // The device-captured track is what this factory sends. - RTCRtpSender sender = peerConnection.addTrack(deviceTrack, + RTCRtpSender customSender = peerConnection.addTrack(customTrack, Collections.singletonList("stream0")); - assertNotNull(sender); - - IllegalStateException e = assertThrows(IllegalStateException.class, () -> { - peerConnection.addTrack(customTrack, Collections.singletonList("stream1")); - }); - assertTrue(e.getMessage().contains("AudioDeviceModule"), e.getMessage()); - - // A transceiver that sends is rejected for the same reason. - RTCRtpTransceiverInit sendRecv = new RTCRtpTransceiverInit(); - - assertThrows(IllegalStateException.class, () -> { - peerConnection.addTransceiver(customTrack, sendRecv); - }); - - // A receive-only transceiver never sends the track, so it is allowed. - RTCRtpTransceiverInit recvOnly = new RTCRtpTransceiverInit(); - recvOnly.direction = RTCRtpTransceiverDirection.RECV_ONLY; + AudioTrackSource deviceSource = deviceFactory.createAudioSource(new AudioOptions()); + AudioTrack deviceTrack = deviceFactory.createAudioTrack("deviceTrack", deviceSource); - RTCRtpTransceiver transceiver = peerConnection.addTransceiver(customTrack, recvOnly); + RTCRtpSender deviceSender = peerConnection.addTrack(deviceTrack, + Collections.singletonList("stream1")); - assertNotNull(transceiver); + assertNotNull(customSender); + assertNotNull(deviceSender); - transceiver.dispose(); - sender.dispose(); + deviceSender.dispose(); + customSender.dispose(); peerConnection.close(); deviceTrack.dispose(); customTrack.dispose(); @@ -312,80 +276,11 @@ void sinkFedTrackAddedToDeviceAudioFactoryIsRejected() { } } - @Test - void sinkFedTrackAddedFirstCommitsFactoryToPushedAudio() { - // Adding a track whose audio is pushed settles the question for the - // factory that owns the connection, even though that factory did not - // create the track. A forwarded remote track reaches a factory this way. - AudioDeviceModule receivingModule = new AudioDeviceModule(AudioLayer.kDummyAudio); - AudioDeviceModule sendingModule = new AudioDeviceModule(AudioLayer.kDummyAudio); - PeerConnectionFactory receivingFactory = new PeerConnectionFactory(receivingModule); - PeerConnectionFactory sendingFactory = new PeerConnectionFactory(sendingModule); - CustomAudioSource customSource = new CustomAudioSource(); - - try { - AudioTrack customTrack = sendingFactory.createAudioTrack("customTrack", customSource); - - RTCPeerConnection peerConnection = receivingFactory.createPeerConnection( - new RTCConfiguration(), candidate -> { }); - - RTCRtpSender sender = peerConnection.addTrack(customTrack, - Collections.singletonList("stream0")); - - assertNotNull(sender); - - // The receiving factory now sends pushed audio, so its own device - // capture is off limits. - assertThrows(IllegalStateException.class, () -> { - receivingFactory.createAudioSource(new AudioOptions()); - }); - - sender.dispose(); - peerConnection.close(); - customTrack.dispose(); - } - finally { - customSource.dispose(); - sendingFactory.dispose(); - receivingFactory.dispose(); - sendingModule.dispose(); - receivingModule.dispose(); - } - } - - @Test - void customAudioTrackNullParamsDoNotCommitFactory() { - AudioDeviceModule audioModule = new AudioDeviceModule(AudioLayer.kDummyAudio); - PeerConnectionFactory audioFactory = new PeerConnectionFactory(audioModule); - CustomAudioSource customSource = new CustomAudioSource(); - - try { - assertThrows(NullPointerException.class, () -> { - audioFactory.createAudioTrack(null, customSource); - }); - assertThrows(NullPointerException.class, () -> { - audioFactory.createAudioSource(null); - }); - - // The rejected custom-track call must not have committed the factory - // to pushed audio, so a device-captured source is still allowed. - AudioTrackSource deviceSource = audioFactory.createAudioSource(new AudioOptions()); - assertNotNull(deviceSource); - - deviceSource.dispose(); - } - finally { - customSource.dispose(); - audioFactory.dispose(); - audioModule.dispose(); - } - } - @Test void deviceAudioSourceAfterPeerConnection() { // A peer connection starts the factory's media engine on the worker - // thread the factory creates for itself. Device capture is then switched - // on that thread, which must still work once audio is being sent. + // thread the factory creates for itself. The device is then started and + // stopped on that thread, which must still work once audio is being sent. AudioDeviceModule audioModule = new AudioDeviceModule(AudioLayer.kDummyAudio); PeerConnectionFactory audioFactory = new PeerConnectionFactory(audioModule); diff --git a/webrtc/src/test/java/dev/onvoid/webrtc/media/audio/ToneAudioDeviceModule.java b/webrtc/src/test/java/dev/onvoid/webrtc/media/audio/ToneAudioDeviceModule.java index 2a4aa1b51..4d06b389c 100644 --- a/webrtc/src/test/java/dev/onvoid/webrtc/media/audio/ToneAudioDeviceModule.java +++ b/webrtc/src/test/java/dev/onvoid/webrtc/media/audio/ToneAudioDeviceModule.java @@ -35,6 +35,14 @@ public ToneAudioDeviceModule(int frequencyHz) { initialize(frequencyHz); } + /** + * Returns whether the module records, as the device a factory opens and + * closes would. + * + * @return True while recording. + */ + public native boolean isRecording(); + private native void initialize(int frequencyHz); }