Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
13 changes: 3 additions & 10 deletions docs/guide/audio/custom-audio-source.md
Original file line number Diff line number Diff line change
Expand Up @@ -45,17 +45,10 @@ PeerConnectionFactory factory = new PeerConnectionFactory();
AudioTrack audioTrack = factory.createAudioTrack("audio-track-id", audioSource);
```

::: warning One kind of audio input per factory
A factory sends audio either from its `AudioDeviceModule`, which is what tracks created with `createAudioSource(AudioOptions)` send, or from audio you push. WebRTC feeds device-captured audio into **every** audio sender of a factory, so a sender that is fed both ways is fed by two threads at once. That is a race inside WebRTC, and it aborts the whole process:
::: tip Pushed and device-captured audio side by side
Each audio sender gets the audio of its own track, so one factory, and one peer connection, can send a `CustomAudioSource` track beside a microphone track created with `createAudioSource(AudioOptions)`, and beside a track forwarded from a remote peer.

```
Fatal error in: ../../audio/audio_send_stream.cc
Check failed: !race_checker.RaceDetected()
```

The factory therefore commits to the kind used first and throws an `IllegalStateException` if you ask for the other. The same applies to a track received from a remote peer and forwarded on, since its audio is pushed as well. If you need both, create a second `PeerConnectionFactory`.

While a factory sends pushed audio it never opens the recording device, so no microphone is used. Playout of received audio is unaffected.
The recording device is opened only while a microphone track is sent. A factory that sends only pushed audio never opens it. Playout of received audio is unaffected.
:::

### Pushing Audio Data
Expand Down
2 changes: 1 addition & 1 deletion docs/guide/client-api-design.md
Original file line number Diff line number Diff line change
Expand Up @@ -466,7 +466,7 @@ public interface LocalMedia {
LocalAudio microphone(AudioOptions options); // echo cancellation, noise suppression, gain control
LocalVideo screen(ScreenOptions options); // a screen or a window from devices()
CustomVideo customVideo(String label);
CustomAudio customAudio(String label); // AudioMode.CUSTOM or HEADLESS
CustomAudio customAudio(String label); // any AudioMode
LocalVideo videoTrack(String label, VideoTrackSource source); // over a source of the core API
LocalAudio audioTrack(String label, AudioTrackSource source); // such as a MediaFileSource's
}
Expand Down
10 changes: 5 additions & 5 deletions docs/guide/client-api.md
Original file line number Diff line number Diff line change
Expand Up @@ -72,15 +72,15 @@ Every timeout of the client API, these two as well as the gathering timeout and

### Audio Modes

WebRTC feeds the audio a factory captures into every audio sender of that factory, so captured audio and audio you push cannot be mixed in one runtime. The runtime's `AudioMode` picks one of them up front:
The runtime's `AudioMode` decides whether it may open the microphone and where it plays remote audio. Every mode sends audio pushed through `CustomAudio`; each sender gets the audio of its own track, so a `DEVICES` runtime can send the microphone and pushed audio side by side, also on one peer:

| Mode | Sends | Plays remote audio | Use it for |
|---|---|---|---|
| `DEVICES` | the microphone | to the speakers | calls on a desktop |
| `CUSTOM` | audio pushed through `CustomAudio` | to the speakers | applications that generate or relay audio |
| `DEVICES` | the microphone and audio pushed through `CustomAudio` | to the speakers | calls on a desktop |
| `CUSTOM` | audio pushed through `CustomAudio`; the microphone is never opened | to the speakers | applications that generate or relay audio |
| `HEADLESS` | audio pushed through `CustomAudio` | to `RemoteAudio.onAudio` handlers only | servers, bots and tests, without any audio hardware |

Asking a runtime for the other kind fails at once with a `ConfigurationException` that names the mode, rather than when the first track is added. Two runtimes can run side by side if an application needs both kinds.
Asking a `CUSTOM` or `HEADLESS` runtime for the microphone fails at once with a `ConfigurationException` that names the mode, rather than when the first track is added. The microphone is opened only while one of its tracks is sent.

## Peers and Signaling

Expand Down Expand Up @@ -475,7 +475,7 @@ Sender audioSender = peer.addTrack(microphone);
| `microphone()`, `microphone(AudioOptions)` | The microphone, with echo cancellation, noise suppression and gain control unless `AudioOptions` say otherwise. `DEVICES` mode only. |
| `screen(ScreenOptions)` | A screen or a window, as the [Desktop Video](/guide/video/desktop-capture) guide describes. |
| `customVideo(String)` | Frames you push. |
| `customAudio(String)` | Audio you push. `CUSTOM` and `HEADLESS` modes only. |
| `customAudio(String)` | Audio you push, in every mode. |
| `videoTrack(String, VideoTrackSource)`, `audioTrack(String, AudioTrackSource)` | A source of the core API, such as one the [media module](/guide/media/media-files) feeds. The source stays yours. |

```java
Expand Down
9 changes: 1 addition & 8 deletions docs/guide/media/media-files.md
Original file line number Diff line number Diff line change
Expand Up @@ -87,14 +87,7 @@ if (source.getVideoSource() != null) {
}
```

::: warning A factory sends either pushed audio or captured audio
A `PeerConnectionFactory` fed from this module is sending pushed audio and cannot also send audio captured by its `AudioDeviceModule`. Give such a factory a dummy audio layer, and use a second factory if an application needs both:

```java
AudioDeviceModule audioModule = new AudioDeviceModule(AudioLayer.kDummyAudio);
PeerConnectionFactory factory = new PeerConnectionFactory(audioModule);
```
:::
The audio of this module is pushed into its track, like that of a `CustomAudioSource`, so the same factory and peer connection can send the microphone beside it. A factory that sends only pushed audio never opens the recording device.

## Reading What a Source Contains

Expand Down
16 changes: 0 additions & 16 deletions webrtc-jni/src/main/cpp/include/JNI_PeerConnectionFactory.h

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

8 changes: 8 additions & 0 deletions webrtc-jni/src/main/cpp/include/JNI_ToneAudioDeviceModule.h

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

56 changes: 18 additions & 38 deletions webrtc-jni/src/main/cpp/include/api/ProxyAudioDeviceModule.h
Original file line number Diff line number Diff line change
Expand Up @@ -33,7 +33,8 @@
namespace jni
{
// Wraps the AudioDeviceModule a PeerConnectionFactory is created with and
// forwards every call to it. Its purpose is to gate the device capture path.
// forwards every call to it, except for the recorded audio and the control
// of the recording.
//
// WebRTC's AudioState starts the module's recording as soon as any audio
// send stream starts and fans every recorded frame out to every send stream
Expand All @@ -45,25 +46,21 @@ namespace jni
// AudioSendStream::SendAudioData and aborts the process, and interleaves
// microphone frames with the pushed audio whenever it does not.
//
// Recorded audio therefore never reaches WebRTC's own transport. While
// capture is enabled the proxy hands it to its DeviceAudioCapture, which
// runs it through the factory's AudioProcessing and on to the
// DeviceAudioSources, the sources PeerConnectionFactory.createAudioSource()
// creates. Their tracks' senders receive it through the tracks' sinks, like
// the senders of every other local source. Render calls still go to
// WebRTC, which feeds the same AudioProcessing the playout signal.
// Recorded audio therefore never reaches WebRTC's own transport. The proxy
// hands it to its DeviceAudioCapture, which runs it through the factory's
// AudioProcessing and on to the DeviceAudioSources, the sources
// PeerConnectionFactory.createAudioSource() creates. Their tracks' senders
// receive it through the tracks' sinks, like the senders of every other
// local source, so a factory may send device-captured, pushed and
// forwarded audio side by side. Render calls still go to WebRTC, which
// feeds the same AudioProcessing the playout signal.
//
// While capture is disabled, InitRecording() and StartRecording() fail so
// AudioState never opens the device, and recorded data that still arrives
// from the wrapped module (from a recording the application started through
// the Java AudioDeviceModule, which holds the wrapped module directly) is
// dropped before it reaches WebRTC. Playout is never affected.
//
// Capture starts disabled and is enabled only once the application asks for
// device-captured audio, which it can only do through
// PeerConnectionFactory.createAudioSource(). A factory that sends nothing
// but sink-fed audio therefore never opens the recording device, and one
// that sends no audio at all never opens it either.
// WebRTC's InitRecording(), StartRecording() and StopRecording() do not
// reach the device either: they only tell the DeviceAudioCapture whether
// WebRTC asks for a recording. The capture records only while that holds
// and a device source has a sink, so a factory that sends nothing but
// pushed or forwarded audio never opens the recording device. Playout is
// never affected.
class ProxyAudioDeviceModule : public webrtc::AudioDeviceModule
{
public:
Expand All @@ -73,16 +70,6 @@ namespace jni
explicit ProxyAudioDeviceModule(webrtc::scoped_refptr<webrtc::AudioDeviceModule> delegate);
~ProxyAudioDeviceModule() override;

// Enables or disables the device capture path. May be called from any
// thread. AudioState starts and stops the module's recording on the
// factory's worker thread, so the switch is made there to keep the two
// from interleaving. The factory creates and owns that thread; this
// proxy learns it when WebRTC registers its transport, which it does
// on the worker thread. Until then no send stream can exist, and the
// switch is made on the calling thread.
void SetCaptureEnabled(bool enabled);
bool CaptureEnabled() const;

// Sets the AudioProcessing the factory is built with, which the
// recorded audio runs through. Called before the factory exists.
void SetAudioProcessing(webrtc::scoped_refptr<webrtc::AudioProcessing> processing);
Expand Down Expand Up @@ -170,16 +157,13 @@ namespace jni
std::optional<Stats> GetStats() const override;

private:
void ApplyCaptureEnabled(bool enabled);

// The AudioTransport registered with the wrapped module in place of
// the one WebRTC registers with this proxy. Render calls are always
// forwarded to WebRTC; recorded data goes to the DeviceAudioCapture,
// and only while capture is enabled.
// forwarded to WebRTC; recorded data goes to the DeviceAudioCapture.
class TransportGate : public webrtc::AudioTransport
{
public:
TransportGate(const std::atomic<bool> & captureEnabled, DeviceAudioCapture & capture);
explicit TransportGate(DeviceAudioCapture & capture);
~TransportGate() override = default;

void SetTarget(webrtc::AudioTransport * target);
Expand Down Expand Up @@ -223,19 +207,15 @@ namespace jni
int64_t * ntp_time_ms) override;

private:
const std::atomic<bool> & captureEnabled_;
DeviceAudioCapture & capture_;
std::atomic<webrtc::AudioTransport *> target_;
};

webrtc::scoped_refptr<webrtc::AudioDeviceModule> delegate_;
std::atomic<bool> captureEnabled_;
// Declared before the gate, which refers to it.
const std::shared_ptr<DeviceAudioCapture> deviceCapture_;
TransportGate gate_;
bool gateRegistered_;
// The factory's worker thread, set while WebRTC's transport is registered.
std::atomic<webrtc::Thread *> workerThread_;
};
}

Expand Down
60 changes: 56 additions & 4 deletions webrtc-jni/src/main/cpp/include/media/audio/DeviceAudioSource.h
Original file line number Diff line number Diff line change
Expand Up @@ -21,12 +21,15 @@
#include "api/audio/audio_processing.h"
#include "api/audio_options.h"
#include "api/media_stream_interface.h"
#include "api/audio/audio_device.h"
#include "api/notifier.h"
#include "api/scoped_refptr.h"
#include "common_audio/resampler/include/push_resampler.h"
#include "rtc_base/synchronization/mutex.h"
#include "rtc_base/thread.h"
#include "rtc_base/thread_annotations.h"

#include <atomic>
#include <cstddef>
#include <cstdint>
#include <memory>
Expand All @@ -48,22 +51,46 @@ namespace jni
// side is left to WebRTC, which feeds the same AudioProcessing the render
// signal, so echo cancellation sees the audio it has to remove.
//
// It also decides when the device records: while a source has a sink and
// WebRTC asks for a recording, which it does while any send stream of the
// factory runs and the application has not switched recording off. A
// factory that sends only pushed or forwarded audio therefore never opens
// the recording device. The device is started and stopped on the factory's
// worker thread, where WebRTC asks for it; a change of the sinks posts the
// update there rather than waiting for it, so that no lock is held while
// the device's recording thread is joined.
//
// Shared by the factory's ProxyAudioDeviceModule, which feeds it on the
// recording thread, and by every DeviceAudioSource it created, which may
// outlive the module.
class DeviceAudioCapture
class DeviceAudioCapture : public std::enable_shared_from_this<DeviceAudioCapture>
{
public:
DeviceAudioCapture() = default;
explicit DeviceAudioCapture(webrtc::scoped_refptr<webrtc::AudioDeviceModule> device);
DeviceAudioCapture(const DeviceAudioCapture &) = delete;
DeviceAudioCapture & operator=(const DeviceAudioCapture &) = delete;

// Sets the AudioProcessing the factory was built with. Called once,
// before the factory exists.
void SetAudioProcessing(webrtc::scoped_refptr<webrtc::AudioProcessing> processing);

// Releases the AudioProcessing and every source, once the module is
// gone and no more audio can arrive.
// Sets the factory's worker thread, while WebRTC's transport is
// registered with the device, and clears it when the transport is
// unregistered, before the thread is stopped. Called on that thread.
void SetWorkerThread(webrtc::Thread * workerThread);

// Records whether WebRTC asks the device to record. Called on the
// worker thread.
void SetRecordingRequested(bool requested);

// Whether WebRTC asks the device to record, which is what WebRTC
// is told when it asks whether the device records, whatever the
// sources: it decides from that when to ask again.
bool RecordingRequested() const;

// Stops a recording this object started, and releases the
// AudioProcessing, the device and every source, once the module is
// gone, its worker thread with it, and no more audio can arrive.
void Detach();

// Called on the recording thread with 10 ms of interleaved 16-bit
Expand All @@ -74,10 +101,31 @@ namespace jni

// A source is registered while it has at least one sink, and held
// for that long, so it is alive whenever audio is handed to it.
// Neither call starts or stops the device: once its lock is
// released, the source calls UpdateRecording().
void AddSource(webrtc::scoped_refptr<DeviceAudioSource> source);
webrtc::scoped_refptr<DeviceAudioSource> RemoveSource(DeviceAudioSource * source);

// Brings the device's recording in line with the sources and with
// WebRTC's request: at once on the worker thread, else posted to it.
// Must not be called while a source's lock is held.
void UpdateRecording();

private:
// Starts or stops the device. Runs on the worker thread.
void ApplyRecording();

bool HasSources();

// Guards the device and the worker thread.
webrtc::Mutex controlMutex_;
webrtc::scoped_refptr<webrtc::AudioDeviceModule> device_ RTC_GUARDED_BY(controlMutex_);
webrtc::Thread * workerThread_ RTC_GUARDED_BY(controlMutex_) = nullptr;

// Set on the worker thread; read by Detach() once it is gone.
std::atomic<bool> recordingRequested_ { false };
std::atomic<bool> recordingStarted_ { false };

// Guards the processing state, which only the recording thread
// uses; it changes threads when the device is restarted.
webrtc::Mutex processMutex_;
Expand Down Expand Up @@ -115,6 +163,10 @@ namespace jni
void Deliver(const webrtc::AudioFrame & frame, std::optional<int64_t> absoluteCaptureTimeMs);

private:
// Adds the sink and registers this source with the capture if it
// is the first one, under the lock.
void AddSinkLocked(webrtc::AudioTrackSinkInterface * sink, bool & registered);

const webrtc::AudioOptions options_;
const std::shared_ptr<DeviceAudioCapture> capture_;

Expand Down
Loading
Loading