diff --git a/docs/guide/audio/custom-audio-source.md b/docs/guide/audio/custom-audio-source.md index 333e4e42b..01ad7c764 100644 --- a/docs/guide/audio/custom-audio-source.md +++ b/docs/guide/audio/custom-audio-source.md @@ -45,17 +45,10 @@ PeerConnectionFactory factory = new PeerConnectionFactory(); AudioTrack audioTrack = factory.createAudioTrack("audio-track-id", audioSource); ``` -::: warning One kind of audio input per factory -A factory sends audio either from its `AudioDeviceModule`, which is what tracks created with `createAudioSource(AudioOptions)` send, or from audio you push. WebRTC feeds device-captured audio into **every** audio sender of a factory, so a sender that is fed both ways is fed by two threads at once. That is a race inside WebRTC, and it aborts the whole process: +::: tip Pushed and device-captured audio side by side +Each audio sender gets the audio of its own track, so one factory, and one peer connection, can send a `CustomAudioSource` track beside a microphone track created with `createAudioSource(AudioOptions)`, and beside a track forwarded from a remote peer. -``` -Fatal error in: ../../audio/audio_send_stream.cc -Check failed: !race_checker.RaceDetected() -``` - -The factory therefore commits to the kind used first and throws an `IllegalStateException` if you ask for the other. The same applies to a track received from a remote peer and forwarded on, since its audio is pushed as well. If you need both, create a second `PeerConnectionFactory`. - -While a factory sends pushed audio it never opens the recording device, so no microphone is used. Playout of received audio is unaffected. +The recording device is opened only while a microphone track is sent. A factory that sends only pushed audio never opens it. Playout of received audio is unaffected. ::: ### Pushing Audio Data diff --git a/docs/guide/client-api-design.md b/docs/guide/client-api-design.md index 12e681c7b..e435c281a 100644 --- a/docs/guide/client-api-design.md +++ b/docs/guide/client-api-design.md @@ -466,7 +466,7 @@ public interface LocalMedia { LocalAudio microphone(AudioOptions options); // echo cancellation, noise suppression, gain control LocalVideo screen(ScreenOptions options); // a screen or a window from devices() CustomVideo customVideo(String label); - CustomAudio customAudio(String label); // AudioMode.CUSTOM or HEADLESS + CustomAudio customAudio(String label); // any AudioMode LocalVideo videoTrack(String label, VideoTrackSource source); // over a source of the core API LocalAudio audioTrack(String label, AudioTrackSource source); // such as a MediaFileSource's } diff --git a/docs/guide/client-api.md b/docs/guide/client-api.md index 879f629aa..df64ba08c 100644 --- a/docs/guide/client-api.md +++ b/docs/guide/client-api.md @@ -72,15 +72,15 @@ Every timeout of the client API, these two as well as the gathering timeout and ### Audio Modes -WebRTC feeds the audio a factory captures into every audio sender of that factory, so captured audio and audio you push cannot be mixed in one runtime. The runtime's `AudioMode` picks one of them up front: +The runtime's `AudioMode` decides whether it may open the microphone and where it plays remote audio. Every mode sends audio pushed through `CustomAudio`; each sender gets the audio of its own track, so a `DEVICES` runtime can send the microphone and pushed audio side by side, also on one peer: | Mode | Sends | Plays remote audio | Use it for | |---|---|---|---| -| `DEVICES` | the microphone | to the speakers | calls on a desktop | -| `CUSTOM` | audio pushed through `CustomAudio` | to the speakers | applications that generate or relay audio | +| `DEVICES` | the microphone and audio pushed through `CustomAudio` | to the speakers | calls on a desktop | +| `CUSTOM` | audio pushed through `CustomAudio`; the microphone is never opened | to the speakers | applications that generate or relay audio | | `HEADLESS` | audio pushed through `CustomAudio` | to `RemoteAudio.onAudio` handlers only | servers, bots and tests, without any audio hardware | -Asking a runtime for the other kind fails at once with a `ConfigurationException` that names the mode, rather than when the first track is added. Two runtimes can run side by side if an application needs both kinds. +Asking a `CUSTOM` or `HEADLESS` runtime for the microphone fails at once with a `ConfigurationException` that names the mode, rather than when the first track is added. The microphone is opened only while one of its tracks is sent. ## Peers and Signaling @@ -475,7 +475,7 @@ Sender audioSender = peer.addTrack(microphone); | `microphone()`, `microphone(AudioOptions)` | The microphone, with echo cancellation, noise suppression and gain control unless `AudioOptions` say otherwise. `DEVICES` mode only. | | `screen(ScreenOptions)` | A screen or a window, as the [Desktop Video](/guide/video/desktop-capture) guide describes. | | `customVideo(String)` | Frames you push. | -| `customAudio(String)` | Audio you push. `CUSTOM` and `HEADLESS` modes only. | +| `customAudio(String)` | Audio you push, in every mode. | | `videoTrack(String, VideoTrackSource)`, `audioTrack(String, AudioTrackSource)` | A source of the core API, such as one the [media module](/guide/media/media-files) feeds. The source stays yours. | ```java diff --git a/docs/guide/media/media-files.md b/docs/guide/media/media-files.md index dfea4e7cb..8c3967fb5 100644 --- a/docs/guide/media/media-files.md +++ b/docs/guide/media/media-files.md @@ -87,14 +87,7 @@ if (source.getVideoSource() != null) { } ``` -::: warning A factory sends either pushed audio or captured audio -A `PeerConnectionFactory` fed from this module is sending pushed audio and cannot also send audio captured by its `AudioDeviceModule`. Give such a factory a dummy audio layer, and use a second factory if an application needs both: - -```java -AudioDeviceModule audioModule = new AudioDeviceModule(AudioLayer.kDummyAudio); -PeerConnectionFactory factory = new PeerConnectionFactory(audioModule); -``` -::: +The audio of this module is pushed into its track, like that of a `CustomAudioSource`, so the same factory and peer connection can send the microphone beside it. A factory that sends only pushed audio never opens the recording device. ## Reading What a Source Contains diff --git a/webrtc-jni/src/main/cpp/include/JNI_PeerConnectionFactory.h b/webrtc-jni/src/main/cpp/include/JNI_PeerConnectionFactory.h index 8995ce152..de1f3a4b9 100644 --- a/webrtc-jni/src/main/cpp/include/JNI_PeerConnectionFactory.h +++ b/webrtc-jni/src/main/cpp/include/JNI_PeerConnectionFactory.h @@ -39,22 +39,6 @@ extern "C" { JNIEXPORT void JNICALL Java_dev_onvoid_webrtc_PeerConnectionFactory_removeVoiceActivityListenerInternal (JNIEnv *, jobject, jlong); - /* - * Class: dev_onvoid_webrtc_PeerConnectionFactory - * Method: setDeviceCaptureEnabled - * Signature: (Z)V - */ - JNIEXPORT void JNICALL Java_dev_onvoid_webrtc_PeerConnectionFactory_setDeviceCaptureEnabled - (JNIEnv *, jobject, jboolean); - - /* - * Class: dev_onvoid_webrtc_PeerConnectionFactory - * Method: isSinkFedAudioTrack - * Signature: (Ldev/onvoid/webrtc/media/MediaStreamTrack;)Z - */ - JNIEXPORT jboolean JNICALL Java_dev_onvoid_webrtc_PeerConnectionFactory_isSinkFedAudioTrack - (JNIEnv *, jobject, jobject); - /* * Class: dev_onvoid_webrtc_PeerConnectionFactory * Method: createVideoTrack diff --git a/webrtc-jni/src/main/cpp/include/JNI_ToneAudioDeviceModule.h b/webrtc-jni/src/main/cpp/include/JNI_ToneAudioDeviceModule.h index 5fde6224d..747c386f6 100644 --- a/webrtc-jni/src/main/cpp/include/JNI_ToneAudioDeviceModule.h +++ b/webrtc-jni/src/main/cpp/include/JNI_ToneAudioDeviceModule.h @@ -15,6 +15,14 @@ extern "C" { JNIEXPORT void JNICALL Java_dev_onvoid_webrtc_media_audio_ToneAudioDeviceModule_initialize (JNIEnv *, jobject, jint); + /* + * Class: dev_onvoid_webrtc_media_audio_ToneAudioDeviceModule + * Method: isRecording + * Signature: ()Z + */ + JNIEXPORT jboolean JNICALL Java_dev_onvoid_webrtc_media_audio_ToneAudioDeviceModule_isRecording + (JNIEnv *, jobject); + #ifdef __cplusplus } #endif diff --git a/webrtc-jni/src/main/cpp/include/api/ProxyAudioDeviceModule.h b/webrtc-jni/src/main/cpp/include/api/ProxyAudioDeviceModule.h index e1ca067fa..452426f8a 100644 --- a/webrtc-jni/src/main/cpp/include/api/ProxyAudioDeviceModule.h +++ b/webrtc-jni/src/main/cpp/include/api/ProxyAudioDeviceModule.h @@ -33,7 +33,8 @@ namespace jni { // Wraps the AudioDeviceModule a PeerConnectionFactory is created with and - // forwards every call to it. Its purpose is to gate the device capture path. + // forwards every call to it, except for the recorded audio and the control + // of the recording. // // WebRTC's AudioState starts the module's recording as soon as any audio // send stream starts and fans every recorded frame out to every send stream @@ -45,25 +46,21 @@ namespace jni // AudioSendStream::SendAudioData and aborts the process, and interleaves // microphone frames with the pushed audio whenever it does not. // - // Recorded audio therefore never reaches WebRTC's own transport. While - // capture is enabled the proxy hands it to its DeviceAudioCapture, which - // runs it through the factory's AudioProcessing and on to the - // DeviceAudioSources, the sources PeerConnectionFactory.createAudioSource() - // creates. Their tracks' senders receive it through the tracks' sinks, like - // the senders of every other local source. Render calls still go to - // WebRTC, which feeds the same AudioProcessing the playout signal. + // Recorded audio therefore never reaches WebRTC's own transport. The proxy + // hands it to its DeviceAudioCapture, which runs it through the factory's + // AudioProcessing and on to the DeviceAudioSources, the sources + // PeerConnectionFactory.createAudioSource() creates. Their tracks' senders + // receive it through the tracks' sinks, like the senders of every other + // local source, so a factory may send device-captured, pushed and + // forwarded audio side by side. Render calls still go to WebRTC, which + // feeds the same AudioProcessing the playout signal. // - // While capture is disabled, InitRecording() and StartRecording() fail so - // AudioState never opens the device, and recorded data that still arrives - // from the wrapped module (from a recording the application started through - // the Java AudioDeviceModule, which holds the wrapped module directly) is - // dropped before it reaches WebRTC. Playout is never affected. - // - // Capture starts disabled and is enabled only once the application asks for - // device-captured audio, which it can only do through - // PeerConnectionFactory.createAudioSource(). A factory that sends nothing - // but sink-fed audio therefore never opens the recording device, and one - // that sends no audio at all never opens it either. + // WebRTC's InitRecording(), StartRecording() and StopRecording() do not + // reach the device either: they only tell the DeviceAudioCapture whether + // WebRTC asks for a recording. The capture records only while that holds + // and a device source has a sink, so a factory that sends nothing but + // pushed or forwarded audio never opens the recording device. Playout is + // never affected. class ProxyAudioDeviceModule : public webrtc::AudioDeviceModule { public: @@ -73,16 +70,6 @@ namespace jni explicit ProxyAudioDeviceModule(webrtc::scoped_refptr delegate); ~ProxyAudioDeviceModule() override; - // Enables or disables the device capture path. May be called from any - // thread. AudioState starts and stops the module's recording on the - // factory's worker thread, so the switch is made there to keep the two - // from interleaving. The factory creates and owns that thread; this - // proxy learns it when WebRTC registers its transport, which it does - // on the worker thread. Until then no send stream can exist, and the - // switch is made on the calling thread. - void SetCaptureEnabled(bool enabled); - bool CaptureEnabled() const; - // Sets the AudioProcessing the factory is built with, which the // recorded audio runs through. Called before the factory exists. void SetAudioProcessing(webrtc::scoped_refptr processing); @@ -170,16 +157,13 @@ namespace jni std::optional GetStats() const override; private: - void ApplyCaptureEnabled(bool enabled); - // The AudioTransport registered with the wrapped module in place of // the one WebRTC registers with this proxy. Render calls are always - // forwarded to WebRTC; recorded data goes to the DeviceAudioCapture, - // and only while capture is enabled. + // forwarded to WebRTC; recorded data goes to the DeviceAudioCapture. class TransportGate : public webrtc::AudioTransport { public: - TransportGate(const std::atomic & captureEnabled, DeviceAudioCapture & capture); + explicit TransportGate(DeviceAudioCapture & capture); ~TransportGate() override = default; void SetTarget(webrtc::AudioTransport * target); @@ -223,19 +207,15 @@ namespace jni int64_t * ntp_time_ms) override; private: - const std::atomic & captureEnabled_; DeviceAudioCapture & capture_; std::atomic target_; }; webrtc::scoped_refptr delegate_; - std::atomic captureEnabled_; // Declared before the gate, which refers to it. const std::shared_ptr deviceCapture_; TransportGate gate_; bool gateRegistered_; - // The factory's worker thread, set while WebRTC's transport is registered. - std::atomic workerThread_; }; } diff --git a/webrtc-jni/src/main/cpp/include/media/audio/DeviceAudioSource.h b/webrtc-jni/src/main/cpp/include/media/audio/DeviceAudioSource.h index 3d13b1fc5..b827bf73c 100644 --- a/webrtc-jni/src/main/cpp/include/media/audio/DeviceAudioSource.h +++ b/webrtc-jni/src/main/cpp/include/media/audio/DeviceAudioSource.h @@ -21,12 +21,15 @@ #include "api/audio/audio_processing.h" #include "api/audio_options.h" #include "api/media_stream_interface.h" +#include "api/audio/audio_device.h" #include "api/notifier.h" #include "api/scoped_refptr.h" #include "common_audio/resampler/include/push_resampler.h" #include "rtc_base/synchronization/mutex.h" +#include "rtc_base/thread.h" #include "rtc_base/thread_annotations.h" +#include #include #include #include @@ -48,13 +51,22 @@ namespace jni // side is left to WebRTC, which feeds the same AudioProcessing the render // signal, so echo cancellation sees the audio it has to remove. // + // It also decides when the device records: while a source has a sink and + // WebRTC asks for a recording, which it does while any send stream of the + // factory runs and the application has not switched recording off. A + // factory that sends only pushed or forwarded audio therefore never opens + // the recording device. The device is started and stopped on the factory's + // worker thread, where WebRTC asks for it; a change of the sinks posts the + // update there rather than waiting for it, so that no lock is held while + // the device's recording thread is joined. + // // Shared by the factory's ProxyAudioDeviceModule, which feeds it on the // recording thread, and by every DeviceAudioSource it created, which may // outlive the module. - class DeviceAudioCapture + class DeviceAudioCapture : public std::enable_shared_from_this { public: - DeviceAudioCapture() = default; + explicit DeviceAudioCapture(webrtc::scoped_refptr device); DeviceAudioCapture(const DeviceAudioCapture &) = delete; DeviceAudioCapture & operator=(const DeviceAudioCapture &) = delete; @@ -62,8 +74,23 @@ namespace jni // before the factory exists. void SetAudioProcessing(webrtc::scoped_refptr processing); - // Releases the AudioProcessing and every source, once the module is - // gone and no more audio can arrive. + // Sets the factory's worker thread, while WebRTC's transport is + // registered with the device, and clears it when the transport is + // unregistered, before the thread is stopped. Called on that thread. + void SetWorkerThread(webrtc::Thread * workerThread); + + // Records whether WebRTC asks the device to record. Called on the + // worker thread. + void SetRecordingRequested(bool requested); + + // Whether WebRTC asks the device to record, which is what WebRTC + // is told when it asks whether the device records, whatever the + // sources: it decides from that when to ask again. + bool RecordingRequested() const; + + // Stops a recording this object started, and releases the + // AudioProcessing, the device and every source, once the module is + // gone, its worker thread with it, and no more audio can arrive. void Detach(); // Called on the recording thread with 10 ms of interleaved 16-bit @@ -74,10 +101,31 @@ namespace jni // A source is registered while it has at least one sink, and held // for that long, so it is alive whenever audio is handed to it. + // Neither call starts or stops the device: once its lock is + // released, the source calls UpdateRecording(). void AddSource(webrtc::scoped_refptr source); webrtc::scoped_refptr RemoveSource(DeviceAudioSource * source); + // Brings the device's recording in line with the sources and with + // WebRTC's request: at once on the worker thread, else posted to it. + // Must not be called while a source's lock is held. + void UpdateRecording(); + private: + // Starts or stops the device. Runs on the worker thread. + void ApplyRecording(); + + bool HasSources(); + + // Guards the device and the worker thread. + webrtc::Mutex controlMutex_; + webrtc::scoped_refptr device_ RTC_GUARDED_BY(controlMutex_); + webrtc::Thread * workerThread_ RTC_GUARDED_BY(controlMutex_) = nullptr; + + // Set on the worker thread; read by Detach() once it is gone. + std::atomic recordingRequested_ { false }; + std::atomic recordingStarted_ { false }; + // Guards the processing state, which only the recording thread // uses; it changes threads when the device is restarted. webrtc::Mutex processMutex_; @@ -115,6 +163,10 @@ namespace jni void Deliver(const webrtc::AudioFrame & frame, std::optional absoluteCaptureTimeMs); private: + // Adds the sink and registers this source with the capture if it + // is the first one, under the lock. + void AddSinkLocked(webrtc::AudioTrackSinkInterface * sink, bool & registered); + const webrtc::AudioOptions options_; const std::shared_ptr capture_; diff --git a/webrtc-jni/src/main/cpp/src/JNI_PeerConnectionFactory.cpp b/webrtc-jni/src/main/cpp/src/JNI_PeerConnectionFactory.cpp index 2c5d0d9ed..9b10c3bc9 100644 --- a/webrtc-jni/src/main/cpp/src/JNI_PeerConnectionFactory.cpp +++ b/webrtc-jni/src/main/cpp/src/JNI_PeerConnectionFactory.cpp @@ -46,7 +46,6 @@ #include "api/rtc_event_log/rtc_event_log_factory.h" #include "api/ProxyAudioDeviceModule.h" -#include "media/audio/CustomAudioSource.h" #include "media/audio/DeviceAudioSource.h" #include "media/audio/codec/AudioCodecFactories.h" #include "media/video/codec/DefaultVideoCodecFactories.h" @@ -147,10 +146,9 @@ JNIEXPORT void JNICALL Java_dev_onvoid_webrtc_PeerConnectionFactory_initialize } } - // Hand WebRTC a proxy of the module so the device capture path can be - // switched off once this factory sends audio from sink-fed sources. See - // ProxyAudioDeviceModule for why both paths must never feed the same - // send stream. + // Hand WebRTC a proxy of the module, which takes the recorded audio to + // the device sources' tracks instead of to every send stream of the + // factory. See ProxyAudioDeviceModule. auto proxy = jni::ProxyAudioDeviceModule::Create(adm); // No worker thread is supplied: WebRTC is removing support for one that @@ -336,26 +334,6 @@ JNIEXPORT void JNICALL Java_dev_onvoid_webrtc_PeerConnectionFactory_removeVoiceA } } -JNIEXPORT void JNICALL Java_dev_onvoid_webrtc_PeerConnectionFactory_setDeviceCaptureEnabled -(JNIEnv * env, jobject caller, jboolean enabled) -{ - jni::CallFromJava(env, [&]() { - // Keeps the module, which a dispose releases with the factory. - jni::NativeLease factory(env, caller); - CHECK_HANDLE(factory); - - jni::ProxyAudioDeviceModule * audioModule = GetHandle(env, caller, "audioModuleHandle"); - CHECK_HANDLE(audioModule); - - try { - audioModule->SetCaptureEnabled(enabled == JNI_TRUE); - } - catch (...) { - ThrowCxxJavaException(env); - } - }); -} - JNIEXPORT jobject JNICALL Java_dev_onvoid_webrtc_PeerConnectionFactory_createAudioSourceInternal (JNIEnv * env, jobject caller, jobject jAudioOptions) { @@ -418,36 +396,6 @@ JNIEXPORT jobject JNICALL Java_dev_onvoid_webrtc_PeerConnectionFactory_createAud } } -JNIEXPORT jboolean JNICALL Java_dev_onvoid_webrtc_PeerConnectionFactory_isSinkFedAudioTrack -(JNIEnv * env, jobject caller, jobject jTrack) -{ - return jni::CallFromJava(env, [&]() -> jboolean { - if (jTrack == nullptr) { - return JNI_FALSE; - } - - jni::NativeLease track(env, jTrack); - - if (track == nullptr || track->kind() != webrtc::MediaStreamTrackInterface::kAudioKind) { - return JNI_FALSE; - } - - webrtc::AudioSourceInterface * source = static_cast(track.get())->GetSource(); - - if (source == nullptr) { - return JNI_FALSE; - } - - // A remote source hands the track's sinks the audio it decodes, and a - // CustomAudioSource hands them the audio the application pushes. Either way - // a sender of that track is fed through the track rather than by the audio - // device module. - bool sinkFed = source->remote() || dynamic_cast(source) != nullptr; - - return sinkFed ? JNI_TRUE : JNI_FALSE; - }); -} - JNIEXPORT jobject JNICALL Java_dev_onvoid_webrtc_PeerConnectionFactory_createVideoTrack (JNIEnv * env, jobject caller, jstring jlabel, jobject jsource) { diff --git a/webrtc-jni/src/main/cpp/src/JNI_ToneAudioDeviceModule.cpp b/webrtc-jni/src/main/cpp/src/JNI_ToneAudioDeviceModule.cpp index 9a08ec499..9ef31ce35 100644 --- a/webrtc-jni/src/main/cpp/src/JNI_ToneAudioDeviceModule.cpp +++ b/webrtc-jni/src/main/cpp/src/JNI_ToneAudioDeviceModule.cpp @@ -37,3 +37,18 @@ JNIEXPORT void JNICALL Java_dev_onvoid_webrtc_media_audio_ToneAudioDeviceModule_ SetHandle(env, caller, audioModule.release()); }); } + +JNIEXPORT jboolean JNICALL Java_dev_onvoid_webrtc_media_audio_ToneAudioDeviceModule_isRecording +(JNIEnv * env, jobject caller) +{ + jboolean recording = JNI_FALSE; + + jni::CallFromJava(env, [&]() { + jni::NativeLease audioModule(env, caller); + CHECK_HANDLE(audioModule); + + recording = audioModule->Recording() ? JNI_TRUE : JNI_FALSE; + }); + + return recording; +} diff --git a/webrtc-jni/src/main/cpp/src/api/ProxyAudioDeviceModule.cpp b/webrtc-jni/src/main/cpp/src/api/ProxyAudioDeviceModule.cpp index 364f63288..01c3d3c91 100644 --- a/webrtc-jni/src/main/cpp/src/api/ProxyAudioDeviceModule.cpp +++ b/webrtc-jni/src/main/cpp/src/api/ProxyAudioDeviceModule.cpp @@ -32,16 +32,19 @@ namespace jni ProxyAudioDeviceModule::ProxyAudioDeviceModule(webrtc::scoped_refptr delegate) : delegate_(std::move(delegate)), - captureEnabled_(false), - deviceCapture_(std::make_shared()), - gate_(captureEnabled_, *deviceCapture_), - gateRegistered_(false), - workerThread_(nullptr) + deviceCapture_(std::make_shared(delegate_)), + gate_(*deviceCapture_), + gateRegistered_(false) { } ProxyAudioDeviceModule::~ProxyAudioDeviceModule() { + // Stops a recording the capture started, which the module would keep + // running, and with it its callback, and lets the sources live on + // without audio. + deviceCapture_->Detach(); + // WebRTC unregisters its transport before releasing the factory. Should // the wrapped module still point at the gate, detach it so the module // does not keep a dangling pointer once this proxy is gone. A module @@ -51,44 +54,6 @@ namespace jni RTC_LOG(LS_WARNING) << "ProxyAudioDeviceModule: the AudioDeviceModule kept its audio callback. " "Stop its playout and recording before disposing the PeerConnectionFactory."; } - - // No more audio is recorded for the sources, which may live on. - deviceCapture_->Detach(); - } - - void ProxyAudioDeviceModule::SetCaptureEnabled(bool enabled) - { - webrtc::Thread * workerThread = workerThread_.load(std::memory_order_acquire); - - if (workerThread) { - workerThread->BlockingCall([this, enabled]() { - ApplyCaptureEnabled(enabled); - }); - } - else { - ApplyCaptureEnabled(enabled); - } - } - - void ProxyAudioDeviceModule::ApplyCaptureEnabled(bool enabled) - { - captureEnabled_.store(enabled, std::memory_order_release); - - RTC_LOG(LS_INFO) << "ProxyAudioDeviceModule: device capture " << (enabled ? "enabled" : "disabled"); - - if (!enabled && delegate_->Recording()) { - // A recording that is already running was started by the application - // through the Java AudioDeviceModule, which holds the wrapped module - // and does not pass through this proxy. Leave it running; its frames - // are dropped instead. WebRTC cannot have started it, because it only - // starts a recording while capture is enabled. - RTC_LOG(LS_INFO) << "ProxyAudioDeviceModule: dropping recorded audio, send streams are sink-fed"; - } - } - - bool ProxyAudioDeviceModule::CaptureEnabled() const - { - return captureEnabled_.load(std::memory_order_acquire); } void ProxyAudioDeviceModule::SetAudioProcessing(webrtc::scoped_refptr processing) @@ -108,16 +73,26 @@ namespace jni int32_t ProxyAudioDeviceModule::RegisterAudioCallback(webrtc::AudioTransport * audioCallback) { + // WebRtcVoiceEngine registers and unregisters its transport on the + // factory's worker thread, which lives exactly as long as the + // transport is registered. + if (audioCallback == nullptr) { + // The factory goes away. Stop a recording the capture started while + // the worker thread still runs; the module rejects the transport's + // removal while it records. + deviceCapture_->SetRecordingRequested(false); + deviceCapture_->SetWorkerThread(nullptr); + } + gate_.SetTarget(audioCallback); int32_t result = delegate_->RegisterAudioCallback(audioCallback ? &gate_ : nullptr); if (result == 0) { gateRegistered_ = audioCallback != nullptr; - - // WebRtcVoiceEngine registers and unregisters its transport on the - // factory's worker thread. - workerThread_.store(audioCallback ? webrtc::Thread::Current() : nullptr, std::memory_order_release); + } + if (audioCallback != nullptr) { + deviceCapture_->SetWorkerThread(webrtc::Thread::Current()); } return result; @@ -199,24 +174,13 @@ namespace jni int32_t ProxyAudioDeviceModule::RecordingIsAvailable(bool * available) { - if (!CaptureEnabled()) { - if (!available) { - return -1; - } - *available = false; - return 0; - } - return delegate_->RecordingIsAvailable(available); } int32_t ProxyAudioDeviceModule::InitRecording() { - if (!CaptureEnabled()) { - return -1; - } - - return delegate_->InitRecording(); + // The capture initializes the recording once it starts it. + return 0; } bool ProxyAudioDeviceModule::RecordingIsInitialized() const @@ -241,21 +205,28 @@ namespace jni int32_t ProxyAudioDeviceModule::StartRecording() { - if (!CaptureEnabled()) { - return -1; - } + // WebRTC asks for a recording whenever a send stream starts, also one + // fed by a pushed or forwarded track. The device records only while a + // device source has a sink as well. + deviceCapture_->SetRecordingRequested(true); - return delegate_->StartRecording(); + return 0; } int32_t ProxyAudioDeviceModule::StopRecording() { - return delegate_->StopRecording(); + // The last send stream stopped, or the application switched recording + // off through RTCPeerConnection.setAudioRecording(false). + deviceCapture_->SetRecordingRequested(false); + + return 0; } bool ProxyAudioDeviceModule::Recording() const { - return delegate_->Recording(); + // What WebRTC asked for, not what the device does: AudioState asks for + // a recording only while it believes none runs. + return deviceCapture_->RecordingRequested(); } int32_t ProxyAudioDeviceModule::InitSpeaker() @@ -435,9 +406,7 @@ namespace jni // TransportGate - ProxyAudioDeviceModule::TransportGate::TransportGate(const std::atomic & captureEnabled, - DeviceAudioCapture & capture) : - captureEnabled_(captureEnabled), + ProxyAudioDeviceModule::TransportGate::TransportGate(DeviceAudioCapture & capture) : capture_(capture), target_(nullptr) { @@ -478,12 +447,6 @@ namespace jni { newMicLevel = currentMicLevel; - if (!captureEnabled_.load(std::memory_order_acquire)) { - // Drop the frame. Send streams of this factory are fed by their - // tracks' sources, not by the device. - return 0; - } - // Never handed to WebRTC's transport, which would feed it into every // send stream of the factory. The device sources' tracks take it to // their senders instead. diff --git a/webrtc-jni/src/main/cpp/src/media/audio/DeviceAudioSource.cpp b/webrtc-jni/src/main/cpp/src/media/audio/DeviceAudioSource.cpp index 908895e8f..3b377f0fa 100644 --- a/webrtc-jni/src/main/cpp/src/media/audio/DeviceAudioSource.cpp +++ b/webrtc-jni/src/main/cpp/src/media/audio/DeviceAudioSource.cpp @@ -55,6 +55,11 @@ namespace jni // DeviceAudioCapture + DeviceAudioCapture::DeviceAudioCapture(webrtc::scoped_refptr device) : + device_(std::move(device)) + { + } + void DeviceAudioCapture::SetAudioProcessing(webrtc::scoped_refptr processing) { webrtc::MutexLock lock(&processMutex_); @@ -62,9 +67,125 @@ namespace jni processing_ = std::move(processing); } + void DeviceAudioCapture::SetWorkerThread(webrtc::Thread * workerThread) + { + webrtc::MutexLock lock(&controlMutex_); + + workerThread_ = workerThread; + } + + void DeviceAudioCapture::SetRecordingRequested(bool requested) + { + recordingRequested_.store(requested); + + ApplyRecording(); + } + + bool DeviceAudioCapture::RecordingRequested() const + { + return recordingRequested_.load(); + } + + void DeviceAudioCapture::UpdateRecording() + { + { + webrtc::MutexLock lock(&controlMutex_); + + if (workerThread_ == nullptr) { + // No factory runs on the device: either none was created yet, + // so no source can have a sink, or it is going away, and + // Detach() stops what is left. + return; + } + if (!workerThread_->IsCurrent()) { + // Posted while the lock is held, so the thread cannot be + // cleared and stopped in between. The task holds only a weak + // reference: the sources may let this object go meanwhile. + workerThread_->PostTask([weak = weak_from_this()]() { + if (auto self = weak.lock()) { + self->ApplyRecording(); + } + }); + return; + } + } + + ApplyRecording(); + } + + void DeviceAudioCapture::ApplyRecording() + { + webrtc::scoped_refptr device; + + { + webrtc::MutexLock lock(&controlMutex_); + + device = device_; + } + + if (!device) { + return; + } + + const bool record = recordingRequested_.load() && HasSources(); + bool started = recordingStarted_.load(); + + if (started && !device->Recording()) { + // The application stopped the recording through its Java + // AudioDeviceModule; start it again if it is still needed. + recordingStarted_.store(false); + started = false; + } + + if (record && !started) { + if (device->Recording()) { + // The application started the recording through its Java + // AudioDeviceModule. Its audio reaches the sources already, and + // the recording stays the application's to stop. + return; + } + if (device->InitRecording() != 0 || device->StartRecording() != 0) { + RTC_LOG(LS_ERROR) << "DeviceAudioCapture: starting the recording failed"; + return; + } + + recordingStarted_.store(true); + + RTC_LOG(LS_INFO) << "DeviceAudioCapture: recording started"; + } + else if (!record && started) { + device->StopRecording(); + + recordingStarted_.store(false); + + RTC_LOG(LS_INFO) << "DeviceAudioCapture: recording stopped"; + } + } + + bool DeviceAudioCapture::HasSources() + { + webrtc::MutexLock lock(&sourcesMutex_); + + return !sources_.empty(); + } + void DeviceAudioCapture::Detach() { std::vector> sources; + webrtc::scoped_refptr device; + + { + webrtc::MutexLock lock(&controlMutex_); + + workerThread_ = nullptr; + device = std::move(device_); + } + + // The worker thread is gone, so nothing starts the device anymore. A + // recording this object started would otherwise outlive the factory. + if (device && recordingStarted_.exchange(false)) { + device->StopRecording(); + } { webrtc::MutexLock lock(&processMutex_); @@ -192,6 +313,19 @@ namespace jni } void DeviceAudioSource::AddSink(webrtc::AudioTrackSinkInterface * sink) + { + bool registered = false; + + AddSinkLocked(sink, registered); + + // Outside the lock: stopping the device joins its recording thread, + // which may be waiting for this lock to deliver audio. + if (registered) { + capture_->UpdateRecording(); + } + } + + void DeviceAudioSource::AddSinkLocked(webrtc::AudioTrackSinkInterface * sink, bool & registered) { webrtc::MutexLock lock(&mutex_); @@ -206,6 +340,7 @@ namespace jni // this lock while holding its own. if (sinks_.size() == 1) { capture_->AddSource(webrtc::scoped_refptr(this)); + registered = true; } } @@ -231,6 +366,11 @@ namespace jni released = capture_->RemoveSource(this); } } + + // Outside the lock, see AddSink(). + if (released) { + capture_->UpdateRecording(); + } } void DeviceAudioSource::Deliver(const webrtc::AudioFrame & frame, std::optional absoluteCaptureTimeMs) diff --git a/webrtc-media/README.md b/webrtc-media/README.md index 855d1fb3e..b8f9379f8 100644 --- a/webrtc-media/README.md +++ b/webrtc-media/README.md @@ -37,8 +37,8 @@ decoded pictures to a `CustomVideoSource` and 10 ms chunks to a `CustomAudioSour per frame. Frames are paced in real time and carry their own presentation times, so playback keeps the timing of the file rather than the timing of a thread. -A factory that sends media from this module sends pushed audio, so it cannot also send audio -captured by its `AudioDeviceModule`. Use a separate factory if you need both. +The audio of this module is pushed into its track, so the same factory can send the microphone +beside it. ## FFmpeg diff --git a/webrtc-media/src/main/java/dev/onvoid/webrtc/media/player/MediaFileSource.java b/webrtc-media/src/main/java/dev/onvoid/webrtc/media/player/MediaFileSource.java index 42305cc8a..a26629925 100644 --- a/webrtc-media/src/main/java/dev/onvoid/webrtc/media/player/MediaFileSource.java +++ b/webrtc-media/src/main/java/dev/onvoid/webrtc/media/player/MediaFileSource.java @@ -49,9 +49,8 @@ * {@link #getVideoSource()} and {@link #getAudioSource()} for {@code null} * before making a track, or ask {@link #getInfo()} first. *

- * A factory fed from here is sending pushed audio, so it cannot also send - * audio captured by its {@code AudioDeviceModule}. Use a separate factory if - * an application needs both. + * The audio is pushed into its track, so the same factory may send the + * microphone beside it. *

* Closing this releases the player and both media sources. Dispose of any * tracks made from them first. diff --git a/webrtc-media/src/main/java/dev/onvoid/webrtc/media/player/MediaPlayer.java b/webrtc-media/src/main/java/dev/onvoid/webrtc/media/player/MediaPlayer.java index 449a54dfc..08ff1ca80 100644 --- a/webrtc-media/src/main/java/dev/onvoid/webrtc/media/player/MediaPlayer.java +++ b/webrtc-media/src/main/java/dev/onvoid/webrtc/media/player/MediaPlayer.java @@ -53,9 +53,8 @@ * player.play(); * } *

- * A factory fed by this player is sending pushed audio, so it cannot also send - * audio captured by its {@code AudioDeviceModule}. Use a separate factory if - * an application needs both. + * The audio is pushed into its track, so the same factory may send the + * microphone beside it. * * @author Alex Andres */ diff --git a/webrtc/src/main/java/dev/onvoid/webrtc/PeerConnectionFactory.java b/webrtc/src/main/java/dev/onvoid/webrtc/PeerConnectionFactory.java index 18e9ecb81..3b0e6e29c 100644 --- a/webrtc/src/main/java/dev/onvoid/webrtc/PeerConnectionFactory.java +++ b/webrtc/src/main/java/dev/onvoid/webrtc/PeerConnectionFactory.java @@ -50,21 +50,18 @@ * It provides factory methods for {@link RTCPeerConnection} and audio/video * {@link MediaStreamTrack}s. *

- * A factory sends audio from exactly one kind of input. Either its - * {@link AudioDeviceModuleBase audio device module} captures the audio, which - * is what tracks created from {@link #createAudioSource(AudioOptions)} send, - * or the audio is pushed into the senders, which is what a - * {@link CustomAudioSource} and a track forwarded from a remote peer do. - * WebRTC feeds device-captured audio into every audio sender of a factory, so - * a sender that is fed both ways hits a race inside WebRTC that aborts the - * process. The factory therefore commits to the kind that is used first and - * rejects the other with an {@link IllegalStateException}. Applications that - * need both create a second factory. + * A factory may send audio of every kind side by side, also on one peer + * connection: audio its {@link AudioDeviceModuleBase audio device module} + * captures, which is what tracks created from + * {@link #createAudioSource(AudioOptions)} send, audio pushed through a + * {@link CustomAudioSource}, and tracks forwarded from a remote peer. Each + * sender gets the audio of its own track only. *

- * The recording device is opened only once - * {@link #createAudioSource(AudioOptions)} is called, since that is the only - * way to send device-captured audio. A factory that sends pushed audio, or no - * audio at all, never opens it. Playout of received audio is never affected. + * The recording device is opened only while a track of a source created by + * {@link #createAudioSource(AudioOptions)} is sent, or has a sink, and some + * audio sender of the factory is sending. A factory that sends only pushed or + * forwarded audio, or no audio at all, never opens it. Playout of received + * audio is never affected. *

* The constructors cover the common combinations of modules. A {@link * #builder() builder} sets any of them, including the video codecs. @@ -82,19 +79,6 @@ public class PeerConnectionFactory extends DisposableNativeObject { } } - /** - * The kind of input that provides the audio this factory sends. - */ - private enum AudioInputMode { - - /** The audio device module captures the audio. */ - DEVICE, - - /** The application pushes the audio through a {@link CustomAudioSource}. */ - CUSTOM - - } - @SuppressWarnings("unused") private long networkThreadHandle; @@ -114,12 +98,6 @@ private enum AudioInputMode { */ private final Map voiceActivityListeners = new IdentityHashMap<>(); - /** Guards {@link #audioInputMode}. */ - private final Object audioInputLock = new Object(); - - /** Set the first time either kind of audio input is used; never reset. */ - private AudioInputMode audioInputMode; - /** * Creates an instance of PeerConnectionFactory. @@ -260,23 +238,16 @@ public static Builder builder() { * The audio reaches the senders of these tracks through the tracks, after * the factory's audio processing: echo cancellation, noise suppression and * gain control as the options ask for. A sink added to such a track gets - * the same audio, on WebRTC's capture thread, while the device records. - *

- * Calling this commits the factory to device-captured audio and lets it open - * the recording device; see the class description. + * the same audio, on WebRTC's capture thread, while the device records; + * see the class description for when it does. * * @param options Audio options to control the audio processing. * * @return The created audio source. - * - * @throws IllegalStateException If this factory already sends audio from a - * {@link CustomAudioSource}. */ public AudioTrackSource createAudioSource(AudioOptions options) { Objects.requireNonNull(options, "AudioOptions is null"); - requireAudioInputMode(AudioInputMode.DEVICE); - return createAudioSourceInternal(options); } @@ -285,87 +256,19 @@ public AudioTrackSource createAudioSource(AudioOptions options) { * {@link RTCPeerConnection} using the {@link RTCPeerConnection#addTrack * addTrack} or {@link RTCPeerConnection#addTransceiver addTransceiver} * methods. - *

- * Passing a {@link CustomAudioSource} commits the factory to pushed audio; - * see the class description. * * @param label The identifier string of the audio track. * @param source The audio source that provides audio data. * * @return The created audio track. - * - * @throws IllegalStateException If the source is a {@link CustomAudioSource} - * and this factory already sends audio - * captured by its audio device module. */ public AudioTrack createAudioTrack(String label, AudioTrackSource source) { Objects.requireNonNull(label, "Audio track label is null"); Objects.requireNonNull(source, "AudioTrackSource is null"); - if (source instanceof CustomAudioSource) { - requireAudioInputMode(AudioInputMode.CUSTOM); - } - return createAudioTrackInternal(label, source); } - /** - * Commits this factory to the given kind of audio input, or verifies that - * it is already committed to it. Committing to {@link AudioInputMode#CUSTOM} - * switches off capture from the audio device module. - * - * @param mode The kind of audio input about to be used. - * - * @throws IllegalStateException If the factory is committed to the other - * kind. - */ - private void requireAudioInputMode(AudioInputMode mode) { - synchronized (audioInputLock) { - if (audioInputMode == mode) { - return; - } - if (audioInputMode == AudioInputMode.CUSTOM) { - throw new IllegalStateException("This PeerConnectionFactory already sends audio pushed through a " - + "CustomAudioSource. WebRTC feeds the audio captured by a factory's AudioDeviceModule into " - + "every audio sender of that factory, so device-captured and custom audio sources cannot be " - + "combined in one factory. Create a separate PeerConnectionFactory for device-captured audio."); - } - if (audioInputMode == AudioInputMode.DEVICE) { - throw new IllegalStateException("This PeerConnectionFactory already sends audio captured by its " - + "AudioDeviceModule. WebRTC feeds that audio into every audio sender of the factory, so " - + "device-captured and custom audio sources cannot be combined in one factory. Create a " - + "separate PeerConnectionFactory for CustomAudioSource tracks."); - } - - // The recording device stays closed until the application asks for - // device-captured audio, which is what keeps captured frames away - // from senders fed by a custom source or by a forwarded remote track. - setDeviceCaptureEnabled(mode == AudioInputMode.DEVICE); - - audioInputMode = mode; - } - } - - /** - * Commits this factory to pushed audio if the given track is an audio track - * whose audio is pushed into its sender rather than captured by the audio - * device module. Tracks backed by a {@link CustomAudioSource} and tracks - * received from a remote peer and forwarded on are of that kind. Called - * before a track becomes a sender of one of this factory's peer - * connections; any other track is ignored. - * - * @param track The track about to be sent, may be {@code null}. - * - * @throws IllegalStateException If the track is of that kind and this - * factory already sends audio captured by its - * audio device module. - */ - void commitAudioInput(MediaStreamTrack track) { - if (track != null && isSinkFedAudioTrack(track)) { - requireAudioInputMode(AudioInputMode.CUSTOM); - } - } - /** * Creates a new {@link VideoTrack}. The video track can be added to the * {@link RTCPeerConnection} using the {@link RTCPeerConnection#addTrack @@ -394,8 +297,7 @@ public RTCPeerConnection createPeerConnection(RTCConfiguration config, observer); if (peerConnection != null) { - // The connection reports back which kind of audio its senders are - // about to send, so this factory can reject a mix of both kinds. + // A close of the connection keeps the factory's threads. peerConnection.setFactory(this); } @@ -534,27 +436,6 @@ private native void initialize(Map fieldTrials, private native RTCPeerConnection createPeerConnectionInternal( RTCConfiguration config, PeerConnectionObserver observer); - /** - * Returns whether the given track is an audio track whose audio reaches a - * sender through the track's sinks instead of through the audio device - * module. That is the case for a {@link CustomAudioSource} and for a track - * received from a remote peer. - * - * @param track The track to classify, must not be {@code null}. - * - * @return True if the track is audio and its source feeds sinks. - */ - private native boolean isSinkFedAudioTrack(MediaStreamTrack track); - - /** - * Enables or disables the capture path of the audio device module WebRTC - * uses for this factory. While disabled, WebRTC cannot start a recording - * and recorded audio never reaches the factory's audio senders. - * - * @param enabled Whether device-captured audio may reach the audio senders. - */ - private native void setDeviceCaptureEnabled(boolean enabled); - /** * Builds a {@link PeerConnectionFactory}. Each module that is not set is diff --git a/webrtc/src/main/java/dev/onvoid/webrtc/RTCPeerConnection.java b/webrtc/src/main/java/dev/onvoid/webrtc/RTCPeerConnection.java index 7719c87a5..6974ce0c0 100644 --- a/webrtc/src/main/java/dev/onvoid/webrtc/RTCPeerConnection.java +++ b/webrtc/src/main/java/dev/onvoid/webrtc/RTCPeerConnection.java @@ -16,9 +16,6 @@ package dev.onvoid.webrtc; -import static java.util.Objects.isNull; -import static java.util.Objects.nonNull; - import dev.onvoid.webrtc.internal.NativeObject; import dev.onvoid.webrtc.media.MediaStreamTrack; import dev.onvoid.webrtc.media.MediaType; @@ -42,9 +39,11 @@ public class RTCPeerConnection extends NativeObject { private long observerHandle; /** - * The factory that created this connection. Set right after construction; - * null only if a connection was obtained some other way. + * The factory that created this connection, whose threads a close keeps + * while it is handed on. Set right after construction; null only if a + * connection was obtained some other way. */ + @SuppressWarnings("unused") private PeerConnectionFactory factory; @@ -56,8 +55,7 @@ private RTCPeerConnection() { } /** - * Sets the factory that created this connection, so that tracks added to it - * can be reported back to that factory. + * Sets the factory that created this connection. * * @param factory The creating factory. */ @@ -101,19 +99,9 @@ void setFactory(PeerConnectionFactory factory) { * to. * * @return The RTCRtpSender which will be used to transmit the media data. - * - * @throws IllegalStateException If the track is an audio track that pushes - * its audio, for example one received from a - * remote peer, while the creating factory - * already sends audio captured by its audio - * device module. See {@link - * PeerConnectionFactory} for why a factory - * sends only one kind of audio input. */ public RTCRtpSender addTrack(MediaStreamTrack track, List streamIds) { - commitAudioInput(track); - return addTrackInternal(track, streamIds); } @@ -139,26 +127,9 @@ public RTCRtpSender addTrack(MediaStreamTrack track, * * @return The RTCRtpTransceiver which will be used to transmit and receive * the media data. - * - * @throws IllegalStateException If the transceiver sends and the track is an - * audio track that pushes its audio, for - * example one received from a remote peer, - * while the creating factory already sends - * audio captured by its audio device module. - * See {@link PeerConnectionFactory} for why a - * factory sends only one kind of audio input. */ public RTCRtpTransceiver addTransceiver(MediaStreamTrack track, RTCRtpTransceiverInit init) { - // A transceiver that only receives never sends the track's audio. Its - // direction can be changed later through RTCRtpTransceiver.setDirection, - // which is not covered here. - if (isNull(init) || isNull(init.direction) - || init.direction == RTCRtpTransceiverDirection.SEND_RECV - || init.direction == RTCRtpTransceiverDirection.SEND_ONLY) { - commitAudioInput(track); - } - return addTransceiverInternal(track, init); } @@ -179,19 +150,6 @@ public RTCRtpTransceiver addTransceiver(MediaType mediaType, RTCRtpTransceiverIn return addTransceiverOfType(mediaType, init); } - /** - * Reports a track that is about to be sent to the factory that created this - * connection, which rejects a track whose audio is pushed while the factory - * already sends audio captured by its audio device module. - * - * @param track The track about to be sent, may be {@code null}. - */ - private void commitAudioInput(MediaStreamTrack track) { - if (nonNull(factory)) { - factory.commitAudioInput(track); - } - } - /** * Creates a new RTCDataChannel object with the given label. The * RTCDataChannelInit dictionary can be used to configure properties of the diff --git a/webrtc/src/main/java/dev/onvoid/webrtc/client/AudioMode.java b/webrtc/src/main/java/dev/onvoid/webrtc/client/AudioMode.java index 2b3f9205b..62ee81e7b 100644 --- a/webrtc/src/main/java/dev/onvoid/webrtc/client/AudioMode.java +++ b/webrtc/src/main/java/dev/onvoid/webrtc/client/AudioMode.java @@ -17,17 +17,22 @@ package dev.onvoid.webrtc.client; /** - * Where the audio a runtime sends comes from. WebRTC feeds the audio a factory - * captures into every audio sender of that factory, so captured and pushed - * audio cannot be mixed within one runtime, and the choice is made when it is - * built. + * Where the audio a runtime sends can come from, and where it plays the audio + * it receives. The choice is made when the runtime is built. */ public enum AudioMode { - /** Capture the microphone and play to the speakers. */ + /** + * Capture the microphone and play to the speakers. Audio pushed through + * {@link CustomAudio} can be sent as well, beside the microphone, also on + * one peer. + */ DEVICES, - /** Send audio pushed through {@link CustomAudio} and play remote audio to the speakers. */ + /** + * Send audio pushed through {@link CustomAudio} and play remote audio to the + * speakers. The microphone is never opened. + */ CUSTOM, /** diff --git a/webrtc/src/main/java/dev/onvoid/webrtc/client/LocalMedia.java b/webrtc/src/main/java/dev/onvoid/webrtc/client/LocalMedia.java index c2789235e..f6d2b9afe 100644 --- a/webrtc/src/main/java/dev/onvoid/webrtc/client/LocalMedia.java +++ b/webrtc/src/main/java/dev/onvoid/webrtc/client/LocalMedia.java @@ -124,9 +124,8 @@ public interface LocalMedia { * * @param label the track label. * @param source the source that feeds the track. A {@link CustomAudioSource} - * needs a runtime built with {@link AudioMode#CUSTOM} or - * {@link AudioMode#HEADLESS}, any other source {@link - * AudioMode#DEVICES}. + * fits any runtime, any other source needs a runtime built + * with {@link AudioMode#DEVICES}. * * @return the track. * @@ -141,9 +140,6 @@ public interface LocalMedia { * @param label the track label. * * @return the track. - * - * @throws ConfigurationException if the runtime's {@link AudioMode} is - * {@link AudioMode#DEVICES}. */ CustomAudio customAudio(String label); diff --git a/webrtc/src/main/java/dev/onvoid/webrtc/client/LocalMediaImpl.java b/webrtc/src/main/java/dev/onvoid/webrtc/client/LocalMediaImpl.java index fbb20616b..38043e46a 100644 --- a/webrtc/src/main/java/dev/onvoid/webrtc/client/LocalMediaImpl.java +++ b/webrtc/src/main/java/dev/onvoid/webrtc/client/LocalMediaImpl.java @@ -153,8 +153,8 @@ public LocalAudio microphone(AudioOptions options) { try (Creation creation = begin()) { if (rtc.audioMode != AudioMode.DEVICES) { throw new ConfigurationException("The runtime was built with AudioMode." + rtc.audioMode - + ", which sends pushed audio only. Build it with AudioMode.DEVICES to capture the microphone, " - + "or use customAudio() to push audio"); + + ", which never opens the microphone. Build it with AudioMode.DEVICES to capture the " + + "microphone, or use customAudio() to push audio"); } AudioTrackSource source = rtc.factory.createAudioSource(options); @@ -242,10 +242,6 @@ public LocalAudio audioTrack(String label, AudioTrackSource source) { try (Creation creation = begin()) { boolean pushed = source instanceof CustomAudioSource; - if (pushed && rtc.audioMode == AudioMode.DEVICES) { - throw new ConfigurationException("A CustomAudioSource needs a runtime built with AudioMode.CUSTOM or " - + "AudioMode.HEADLESS; this one captures the microphone"); - } if (!pushed && rtc.audioMode != AudioMode.DEVICES) { throw new ConfigurationException("A captured audio source needs a runtime built with AudioMode.DEVICES; " + "this one was built with AudioMode." + rtc.audioMode); @@ -263,12 +259,6 @@ public CustomAudio customAudio(String label) { requireNonNull(label, "label"); try (Creation creation = begin()) { - if (rtc.audioMode == AudioMode.DEVICES) { - throw new ConfigurationException("The runtime was built with AudioMode.DEVICES, which sends the " - + "microphone. WebRTC feeds captured audio into every audio sender of a runtime, so pushed " - + "audio needs a runtime built with AudioMode.CUSTOM or AudioMode.HEADLESS"); - } - CustomAudioSource source = new CustomAudioSource(); try { diff --git a/webrtc/src/test/java/dev/onvoid/webrtc/DeviceAudioSendTests.java b/webrtc/src/test/java/dev/onvoid/webrtc/DeviceAudioSendTests.java index 33886cb7f..3efbee0c7 100644 --- a/webrtc/src/test/java/dev/onvoid/webrtc/DeviceAudioSendTests.java +++ b/webrtc/src/test/java/dev/onvoid/webrtc/DeviceAudioSendTests.java @@ -16,6 +16,7 @@ package dev.onvoid.webrtc; +import static org.junit.jupiter.api.Assertions.assertFalse; import static org.junit.jupiter.api.Assertions.assertNotNull; import static org.junit.jupiter.api.Assertions.assertTrue; @@ -23,6 +24,7 @@ import dev.onvoid.webrtc.media.audio.AudioTrack; import dev.onvoid.webrtc.media.audio.AudioTrackSink; import dev.onvoid.webrtc.media.audio.AudioTrackSource; +import dev.onvoid.webrtc.media.audio.CustomAudioSource; import dev.onvoid.webrtc.media.audio.ToneAudioDeviceModule; import org.junit.jupiter.api.Test; @@ -33,13 +35,16 @@ import java.util.Map; import java.util.concurrent.CountDownLatch; import java.util.concurrent.TimeUnit; +import java.util.concurrent.atomic.AtomicBoolean; import java.util.concurrent.atomic.AtomicReference; +import java.util.function.BooleanSupplier; import java.util.function.Predicate; /** * Audio captured by the factory's audio device module reaches the senders * through the track of a device source, like the audio of every other local - * source. The module here records a tone in place of a microphone. + * source, so a factory sends it beside pushed audio. The module here records a + * tone in place of a microphone. */ class DeviceAudioSendTests extends TestBase { @@ -133,6 +138,116 @@ void echoStatisticsReported() throws Exception { } } + @Test + void deviceAndPushedTracksOnOnePeer() throws Exception { + try (DeviceCall call = new DeviceCall(new AudioOptions(), true, true)) { + call.connect(); + call.startPushing(); + + // The callee's transceivers follow the order of the offer: the + // microphone first, then the pushed silence. + RTCRtpReceiver[] receivers = call.getReceivers(); + AudioTrack microphone = (AudioTrack) receivers[0].getTrack(); + AudioTrack pushed = (AudioTrack) receivers[1].getTrack(); + + CountDownLatch microphoneAudible = new CountDownLatch(5); + CountDownLatch pushedReceived = new CountDownLatch(50); + AtomicBoolean pushedAudible = new AtomicBoolean(); + + AudioTrackSink microphoneSink = (data, bits, rate, channels, frames) -> { + if (isAudible(data, bits, channels, frames)) { + microphoneAudible.countDown(); + } + }; + // Recorded audio fed into the pushed track's sender as well would + // make the silence audible. + AudioTrackSink pushedSink = (data, bits, rate, channels, frames) -> { + if (isAudible(data, bits, channels, frames)) { + pushedAudible.set(true); + } + pushedReceived.countDown(); + }; + + microphone.addSink(microphoneSink); + pushed.addSink(pushedSink); + + try { + assertTrue(microphoneAudible.await(TIMEOUT_SECONDS, TimeUnit.SECONDS), + "No recorded audio at the callee"); + assertTrue(pushedReceived.await(TIMEOUT_SECONDS, TimeUnit.SECONDS), + "No pushed audio at the callee"); + assertFalse(pushedAudible.get(), "Recorded audio reached the pushed track"); + } + finally { + microphone.removeSink(microphoneSink); + pushed.removeSink(pushedSink); + + for (RTCRtpReceiver receiver : receivers) { + receiver.dispose(); + } + } + } + } + + @Test + void pushedAudioAloneNeverRecords() throws Exception { + try (DeviceCall call = new DeviceCall(new AudioOptions(), false, true)) { + call.connect(); + call.startPushing(); + + // A send stream runs, so WebRTC asks for a recording, but no device + // source needs it. + assertNotNull(call.awaitStats(call.customSender, RTCStatsType.OUTBOUND_RTP, + attributes -> number(attributes, "packetsSent") > 0), "Nothing was sent"); + assertFalse(call.audioModule.isRecording(), "The device records for pushed audio"); + } + } + + @Test + void recordingFollowsSetAudioRecording() throws Exception { + try (DeviceCall call = new DeviceCall(new AudioOptions(), true, false)) { + call.connect(); + + assertTrue(await(call.audioModule::isRecording), "The device does not record"); + + call.caller.getPeerConnection().setAudioRecording(false); + assertTrue(await(() -> !call.audioModule.isRecording()), "Recording was not switched off"); + + call.caller.getPeerConnection().setAudioRecording(true); + assertTrue(await(call.audioModule::isRecording), "Recording was not switched on again"); + } + } + + @Test + void recordingStopsWithLastDeviceTrack() throws Exception { + try (DeviceCall call = new DeviceCall(new AudioOptions(), true, true)) { + call.connect(); + call.startPushing(); + + assertTrue(await(call.audioModule::isRecording), "The device does not record"); + + // The pushed track keeps a send stream, and WebRTC's request for a + // recording, alive; the microphone track's sink is gone. + call.caller.getPeerConnection().removeTrack(call.sender); + + assertTrue(await(() -> !call.audioModule.isRecording()), "The device still records"); + } + } + + private static boolean await(BooleanSupplier condition) throws InterruptedException { + long deadline = System.nanoTime() + TimeUnit.SECONDS.toNanos(TIMEOUT_SECONDS); + + while (System.nanoTime() < deadline) { + if (condition.getAsBoolean()) { + return true; + } + + Thread.sleep(20); + } + + return condition.getAsBoolean(); + } + private static boolean isAudible(byte[] data, int bits, int channels, int frames) { if (bits != 16) { return false; @@ -159,8 +274,9 @@ private static double number(Map attributes, String name) { /** - * A call that sends the track of a device source from the caller to the - * callee, both on a factory whose audio device module records a tone. + * A call that sends the track of a device source, a track of pushed + * silence, or both, from the caller to the callee, all on one factory whose + * audio device module records a tone. */ private static class DeviceCall implements AutoCloseable { @@ -178,12 +294,24 @@ private static class DeviceCall implements AutoCloseable { final RTCRtpSender sender; + final CustomAudioSource customSource; + + final AudioTrack customTrack; + + final RTCRtpSender customSender; + + private volatile boolean pushing; + + private Thread pusher; + DeviceCall(AudioOptions options) { + this(options, true, false); + } + + DeviceCall(AudioOptions options, boolean device, boolean pushed) { audioModule = new ToneAudioDeviceModule(440); factory = new PeerConnectionFactory(audioModule); - source = factory.createAudioSource(options); - track = factory.createAudioTrack("microphone", source); caller = new TestPeerConnection(factory); callee = new TestPeerConnection(factory); @@ -191,7 +319,57 @@ private static class DeviceCall implements AutoCloseable { caller.setRemotePeerConnection(callee); callee.setRemotePeerConnection(caller); - sender = caller.getPeerConnection().addTrack(track, Collections.singletonList("stream")); + if (device) { + source = factory.createAudioSource(options); + track = factory.createAudioTrack("microphone", source); + sender = caller.getPeerConnection().addTrack(track, Collections.singletonList("stream")); + } + else { + source = null; + track = null; + sender = null; + } + + if (pushed) { + customSource = new CustomAudioSource(); + customTrack = factory.createAudioTrack("pushed", customSource); + customSender = caller.getPeerConnection().addTrack(customTrack, Collections.singletonList("stream")); + } + else { + customSource = null; + customTrack = null; + customSender = null; + } + } + + /** + * Starts pushing 10 ms of silence every 10 ms into the custom source. + */ + void startPushing() { + pushing = true; + pusher = new Thread(() -> { + byte[] silence = new byte[480 * 2]; + long next = System.nanoTime(); + + while (pushing) { + customSource.pushAudio(silence, 16, 48000, 1, 480); + + next += 10_000_000L; + + long wait = next - System.nanoTime(); + + if (wait > 0) { + try { + TimeUnit.NANOSECONDS.sleep(wait); + } + catch (InterruptedException e) { + return; + } + } + } + }, "DeviceCall-pusher"); + pusher.setDaemon(true); + pusher.start(); } void connect() throws Exception { @@ -203,31 +381,51 @@ void connect() throws Exception { } /** - * Returns the callee's receiver of the track, which is the caller's to - * dispose. + * Returns the callee's receiver of the first track, which is the + * caller's to dispose. */ RTCRtpReceiver getReceiver() { + RTCRtpReceiver[] receivers = getReceivers(); + + for (int i = 1; i < receivers.length; i++) { + receivers[i].dispose(); + } + + return receivers[0]; + } + + /** + * Returns the callee's receivers, in the order of the offer, which are + * the caller's to dispose. + */ + RTCRtpReceiver[] getReceivers() { RTCRtpTransceiver[] transceivers = callee.getPeerConnection().getTransceivers(); - RTCRtpReceiver receiver = transceivers[0].getReceiver(); + RTCRtpReceiver[] receivers = new RTCRtpReceiver[transceivers.length]; - for (RTCRtpTransceiver transceiver : transceivers) { - transceiver.dispose(); + for (int i = 0; i < transceivers.length; i++) { + receivers[i] = transceivers[i].getReceiver(); + transceivers[i].dispose(); } - return receiver; + return receivers; } Map awaitMediaSourceStats(Predicate> condition) throws InterruptedException { + return awaitStats(sender, RTCStatsType.MEDIA_SOURCE, condition); + } + + Map awaitStats(RTCRtpSender statsSender, RTCStatsType type, + Predicate> condition) throws InterruptedException { long deadline = System.nanoTime() + TimeUnit.SECONDS.toNanos(TIMEOUT_SECONDS); while (System.nanoTime() < deadline) { CountDownLatch done = new CountDownLatch(1); AtomicReference> found = new AtomicReference<>(); - caller.getPeerConnection().getStats(sender, report -> { + caller.getPeerConnection().getStats(statsSender, report -> { for (RTCStats stats : report.getStats().values()) { - if (stats.getType() == RTCStatsType.MEDIA_SOURCE && condition.test(stats.getAttributes())) { + if (stats.getType() == type && condition.test(stats.getAttributes())) { found.set(stats.getAttributes()); } } @@ -247,14 +445,32 @@ Map awaitMediaSourceStats(Predicate> conditi } @Override - public void close() { - sender.dispose(); + public void close() throws InterruptedException { + pushing = false; + + if (pusher != null) { + pusher.join(); + } + + if (sender != null) { + sender.dispose(); + } + if (customSender != null) { + customSender.dispose(); + } caller.close(); callee.close(); - track.dispose(); - source.dispose(); + if (track != null) { + track.dispose(); + source.dispose(); + } + if (customTrack != null) { + customTrack.dispose(); + customSource.dispose(); + } + factory.dispose(); audioModule.dispose(); } diff --git a/webrtc/src/test/java/dev/onvoid/webrtc/PeerConnectionFactoryTests.java b/webrtc/src/test/java/dev/onvoid/webrtc/PeerConnectionFactoryTests.java index 6ec9d781b..1563798cf 100644 --- a/webrtc/src/test/java/dev/onvoid/webrtc/PeerConnectionFactoryTests.java +++ b/webrtc/src/test/java/dev/onvoid/webrtc/PeerConnectionFactoryTests.java @@ -190,61 +190,41 @@ void createAudioTrack() { } @Test - void customAudioSourceAfterDeviceAudioSourceIsRejected() { - // WebRTC feeds device-captured audio into every audio sender of a - // factory. A CustomAudioSource track would be fed twice, which aborts - // the process inside WebRTC, so the factory must refuse it up front. + void deviceAndCustomSourcesShareFactory() { + // Each sender gets the audio of its own track, so a factory sends + // device-captured and pushed audio side by side, also on one connection. AudioDeviceModule audioModule = new AudioDeviceModule(AudioLayer.kDummyAudio); PeerConnectionFactory audioFactory = new PeerConnectionFactory(audioModule); CustomAudioSource customSource = new CustomAudioSource(); try { AudioTrackSource deviceSource = audioFactory.createAudioSource(new AudioOptions()); - - IllegalStateException e = assertThrows(IllegalStateException.class, () -> { - audioFactory.createAudioTrack("customTrack", customSource); - }); - assertTrue(e.getMessage().contains("CustomAudioSource"), e.getMessage()); - - // The device path stays usable. AudioTrack deviceTrack = audioFactory.createAudioTrack("deviceTrack", deviceSource); - assertNotNull(deviceTrack); - - deviceTrack.dispose(); - deviceSource.dispose(); - } - finally { - customSource.dispose(); - audioFactory.dispose(); - audioModule.dispose(); - } - } + AudioTrack customTrack = audioFactory.createAudioTrack("customTrack", customSource); - @Test - void deviceAudioSourceAfterCustomAudioSourceIsRejected() { - AudioDeviceModule audioModule = new AudioDeviceModule(AudioLayer.kDummyAudio); - PeerConnectionFactory audioFactory = new PeerConnectionFactory(audioModule); - CustomAudioSource customSource = new CustomAudioSource(); - CustomAudioSource otherCustomSource = new CustomAudioSource(); + // Either order of creation is fine. + AudioTrackSource otherDeviceSource = audioFactory.createAudioSource(new AudioOptions()); - try { - AudioTrack customTrack = audioFactory.createAudioTrack("customTrack", customSource); - assertNotNull(customTrack); + RTCPeerConnection peerConnection = audioFactory.createPeerConnection( + new RTCConfiguration(), candidate -> { }); - IllegalStateException e = assertThrows(IllegalStateException.class, () -> { - audioFactory.createAudioSource(new AudioOptions()); - }); - assertTrue(e.getMessage().contains("AudioDeviceModule"), e.getMessage()); + RTCRtpSender deviceSender = peerConnection.addTrack(deviceTrack, + Collections.singletonList("stream0")); + RTCRtpTransceiver customTransceiver = peerConnection.addTransceiver(customTrack, + new RTCRtpTransceiverInit()); - // Any number of custom sources may share the factory. - AudioTrack otherCustomTrack = audioFactory.createAudioTrack("otherCustomTrack", otherCustomSource); - assertNotNull(otherCustomTrack); + assertNotNull(deviceSender); + assertNotNull(customTransceiver); - otherCustomTrack.dispose(); + customTransceiver.dispose(); + deviceSender.dispose(); + peerConnection.close(); customTrack.dispose(); + deviceTrack.dispose(); + otherDeviceSource.dispose(); + deviceSource.dispose(); } finally { - otherCustomSource.dispose(); customSource.dispose(); audioFactory.dispose(); audioModule.dispose(); @@ -252,10 +232,10 @@ void deviceAudioSourceAfterCustomAudioSourceIsRejected() { } @Test - void sinkFedTrackAddedToDeviceAudioFactoryIsRejected() { - // A track whose audio is pushed rather than captured, here one backed by - // a CustomAudioSource but equally one forwarded from a remote peer, may - // not become a sender of a factory that sends device-captured audio. + void foreignPushedTrackBesideDeviceTrack() { + // A track whose audio is pushed and that another factory created, as a + // track forwarded from a remote peer is, may be sent beside a + // device-captured one. AudioDeviceModule deviceModule = new AudioDeviceModule(AudioLayer.kDummyAudio); AudioDeviceModule customModule = new AudioDeviceModule(AudioLayer.kDummyAudio); PeerConnectionFactory deviceFactory = new PeerConnectionFactory(deviceModule); @@ -263,41 +243,25 @@ void sinkFedTrackAddedToDeviceAudioFactoryIsRejected() { CustomAudioSource customSource = new CustomAudioSource(); try { - AudioTrackSource deviceSource = deviceFactory.createAudioSource(new AudioOptions()); - AudioTrack deviceTrack = deviceFactory.createAudioTrack("deviceTrack", deviceSource); AudioTrack customTrack = customFactory.createAudioTrack("customTrack", customSource); RTCPeerConnection peerConnection = deviceFactory.createPeerConnection( new RTCConfiguration(), candidate -> { }); - // The device-captured track is what this factory sends. - RTCRtpSender sender = peerConnection.addTrack(deviceTrack, + RTCRtpSender customSender = peerConnection.addTrack(customTrack, Collections.singletonList("stream0")); - assertNotNull(sender); - - IllegalStateException e = assertThrows(IllegalStateException.class, () -> { - peerConnection.addTrack(customTrack, Collections.singletonList("stream1")); - }); - assertTrue(e.getMessage().contains("AudioDeviceModule"), e.getMessage()); - - // A transceiver that sends is rejected for the same reason. - RTCRtpTransceiverInit sendRecv = new RTCRtpTransceiverInit(); - - assertThrows(IllegalStateException.class, () -> { - peerConnection.addTransceiver(customTrack, sendRecv); - }); - - // A receive-only transceiver never sends the track, so it is allowed. - RTCRtpTransceiverInit recvOnly = new RTCRtpTransceiverInit(); - recvOnly.direction = RTCRtpTransceiverDirection.RECV_ONLY; + AudioTrackSource deviceSource = deviceFactory.createAudioSource(new AudioOptions()); + AudioTrack deviceTrack = deviceFactory.createAudioTrack("deviceTrack", deviceSource); - RTCRtpTransceiver transceiver = peerConnection.addTransceiver(customTrack, recvOnly); + RTCRtpSender deviceSender = peerConnection.addTrack(deviceTrack, + Collections.singletonList("stream1")); - assertNotNull(transceiver); + assertNotNull(customSender); + assertNotNull(deviceSender); - transceiver.dispose(); - sender.dispose(); + deviceSender.dispose(); + customSender.dispose(); peerConnection.close(); deviceTrack.dispose(); customTrack.dispose(); @@ -312,80 +276,11 @@ void sinkFedTrackAddedToDeviceAudioFactoryIsRejected() { } } - @Test - void sinkFedTrackAddedFirstCommitsFactoryToPushedAudio() { - // Adding a track whose audio is pushed settles the question for the - // factory that owns the connection, even though that factory did not - // create the track. A forwarded remote track reaches a factory this way. - AudioDeviceModule receivingModule = new AudioDeviceModule(AudioLayer.kDummyAudio); - AudioDeviceModule sendingModule = new AudioDeviceModule(AudioLayer.kDummyAudio); - PeerConnectionFactory receivingFactory = new PeerConnectionFactory(receivingModule); - PeerConnectionFactory sendingFactory = new PeerConnectionFactory(sendingModule); - CustomAudioSource customSource = new CustomAudioSource(); - - try { - AudioTrack customTrack = sendingFactory.createAudioTrack("customTrack", customSource); - - RTCPeerConnection peerConnection = receivingFactory.createPeerConnection( - new RTCConfiguration(), candidate -> { }); - - RTCRtpSender sender = peerConnection.addTrack(customTrack, - Collections.singletonList("stream0")); - - assertNotNull(sender); - - // The receiving factory now sends pushed audio, so its own device - // capture is off limits. - assertThrows(IllegalStateException.class, () -> { - receivingFactory.createAudioSource(new AudioOptions()); - }); - - sender.dispose(); - peerConnection.close(); - customTrack.dispose(); - } - finally { - customSource.dispose(); - sendingFactory.dispose(); - receivingFactory.dispose(); - sendingModule.dispose(); - receivingModule.dispose(); - } - } - - @Test - void customAudioTrackNullParamsDoNotCommitFactory() { - AudioDeviceModule audioModule = new AudioDeviceModule(AudioLayer.kDummyAudio); - PeerConnectionFactory audioFactory = new PeerConnectionFactory(audioModule); - CustomAudioSource customSource = new CustomAudioSource(); - - try { - assertThrows(NullPointerException.class, () -> { - audioFactory.createAudioTrack(null, customSource); - }); - assertThrows(NullPointerException.class, () -> { - audioFactory.createAudioSource(null); - }); - - // The rejected custom-track call must not have committed the factory - // to pushed audio, so a device-captured source is still allowed. - AudioTrackSource deviceSource = audioFactory.createAudioSource(new AudioOptions()); - assertNotNull(deviceSource); - - deviceSource.dispose(); - } - finally { - customSource.dispose(); - audioFactory.dispose(); - audioModule.dispose(); - } - } - @Test void deviceAudioSourceAfterPeerConnection() { // A peer connection starts the factory's media engine on the worker - // thread the factory creates for itself. Device capture is then switched - // on that thread, which must still work once audio is being sent. + // thread the factory creates for itself. The device is then started and + // stopped on that thread, which must still work once audio is being sent. AudioDeviceModule audioModule = new AudioDeviceModule(AudioLayer.kDummyAudio); PeerConnectionFactory audioFactory = new PeerConnectionFactory(audioModule); diff --git a/webrtc/src/test/java/dev/onvoid/webrtc/media/audio/ToneAudioDeviceModule.java b/webrtc/src/test/java/dev/onvoid/webrtc/media/audio/ToneAudioDeviceModule.java index 2a4aa1b51..4d06b389c 100644 --- a/webrtc/src/test/java/dev/onvoid/webrtc/media/audio/ToneAudioDeviceModule.java +++ b/webrtc/src/test/java/dev/onvoid/webrtc/media/audio/ToneAudioDeviceModule.java @@ -35,6 +35,14 @@ public ToneAudioDeviceModule(int frequencyHz) { initialize(frequencyHz); } + /** + * Returns whether the module records, as the device a factory opens and + * closes would. + * + * @return True while recording. + */ + public native boolean isRecording(); + private native void initialize(int frequencyHz); }