Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions docs/guide/client-api-design.md
Original file line number Diff line number Diff line change
Expand Up @@ -340,6 +340,7 @@ public interface Peer extends AutoCloseable {

// Sending. In AUTOMATIC mode each change renegotiates by itself.
Sender addTrack(LocalTrack track, String... streamIds);
Sender addTrack(LocalTrack track, Simulcast simulcast, String... streamIds); // video, two or three layers with rids
List<Sender> senders();

// Receiving. Late subscribers get the tracks that already arrived.
Expand Down Expand Up @@ -635,6 +636,8 @@ public interface Sender {
LocalTrack track();
CompletableFuture<Void> replaceTrack(LocalTrack track); // no renegotiation, null sends nothing
CompletableFuture<Void> configure(Encoding encoding); // max bitrate, max frame rate, scale down, active
CompletableFuture<Void> configure(String rid, Encoding encoding); // one simulcast layer
List<String> rids(); // the simulcast layers, none without
CompletableFuture<Void> transform(RTCEncodedFrameTransformer transform); // null sends frames unchanged
CompletableFuture<Void> generateKeyFrame();
CompletableFuture<Void> remove(); // renegotiates in AUTOMATIC mode
Expand Down
31 changes: 31 additions & 0 deletions docs/guide/client/sending-media.md
Original file line number Diff line number Diff line change
Expand Up @@ -134,6 +134,29 @@ audioSender.sendTones("*99,1", Duration.ofMillis(200), Duration.ofMillis(100));

The characters 0 to 9, A to D, `#` and `*` are tones, and a comma is a pause of 2 seconds. What happens to queued and stopped tones is described under [Details](#dtmf-tones), and the [DTMF Sender](/guide/audio/dtmf-sender) guide has more.

## Simulcast

With simulcast, a video track is sent in several layers at once, each at its own resolution and bitrate. A server that forwards the stream, such as an SFU, then gives each receiver the layer its connection takes. Each layer has an RTP stream id, its `rid`:

```java
Sender sender = peer.addTrack(camera, Simulcast.builder()
.layer("q", Encoding.builder().scaleDownBy(4).maxBitrate(150_000).build())
.layer("h", Encoding.builder().scaleDownBy(2).maxBitrate(500_000).build())
.layer("f", Encoding.builder().maxBitrate(1_500_000).build())
.build());
```

`Simulcast.standard()` gives three layers in full, half and quarter resolution, with the rids `f`, `h` and `q`.

`configure(rid, encoding)` changes one layer, and `configure(encoding)` every layer alike. A server may ask the sender to pause the layers no receiver wants:

```java
sender.configure("f", Encoding.builder().active(false).build()); // stop sending full resolution
sender.configure("f", Encoding.builder().active(true).build()); // and start again
```

`rids()` lists the layers of a sender, in their order.

## Switching Cameras

A `LocalCamera` switches to another camera without renegotiating. Every peer that sends it keeps sending it, and preview handlers keep getting frames:
Expand Down Expand Up @@ -184,6 +207,14 @@ If the thread stalls, as in a long garbage collection pause, it makes up for at
- A video sender, or a sender whose remote peer accepts no telephone events, fails the future with an `IllegalStateException`.
- So do tones that WebRTC stops sending midway, for example because the sender was paused with `configure`. The future then fails a few seconds after their planned end.

### Simulcast Layers

- A simulcast takes two or three layers, the most WebRTC sends.
- A rid has 1 to 16 letters and digits, and is unique among the layers. It cannot change once the track is added.
- Only video tracks take simulcast. `addTrack` throws an `IllegalArgumentException` for an audio track.
- A remote peer without simulcast leaves it out of its answer and gets only the first layer. A plain WebRTC peer, such as another `Peer`, is one of them.
- `configure(rid, encoding)` throws an `IllegalArgumentException` for a rid the sender does not have.

### When a Camera Switch Fails

- If the new camera cannot be opened, the future fails with a `MediaException`, and the track keeps sending the camera it had.
Expand Down
2 changes: 1 addition & 1 deletion docs/guide/client/whip-whep.md
Original file line number Diff line number Diff line change
Expand Up @@ -95,7 +95,7 @@ Whip.Session live = Whip.to(URI.create("https://example.com/live/whip"))
.get(15, TimeUnit.SECONDS);
```

Simulcast needs a video track. `Sender.configure` on its sender changes every layer alike.
Simulcast needs a video track. `Sender.configure(Encoding)` on its sender changes every layer alike, and `configure(rid, Encoding)` one layer. Simulcast outside WHIP is described under [Simulcast](/guide/client/sending-media#simulcast).

## Errors

Expand Down
16 changes: 16 additions & 0 deletions webrtc/src/main/java/dev/onvoid/webrtc/client/Peer.java
Original file line number Diff line number Diff line change
Expand Up @@ -75,6 +75,22 @@ public interface Peer extends AutoCloseable {
*/
Sender addTrack(LocalTrack track, String... streamIds);

/**
* Sends a video track in the layers of the given simulcast, each with
* its rid. Otherwise as {@link #addTrack(LocalTrack, String...)}. Should
* WebRTC refuse the layers, the error handlers of the peer hear of it,
* and every operation of the sender fails.
*
* @param track a video track of the same runtime.
* @param simulcast the layers to send the track in.
* @param streamIds the streams the remote peer sees the track in.
*
* @return the handle of the new sender, at once.
*
* @throws IllegalArgumentException if the track is not a video track.
*/
Sender addTrack(LocalTrack track, Simulcast simulcast, String... streamIds);

/**
* @return the senders that were added and not removed.
*/
Expand Down
30 changes: 29 additions & 1 deletion webrtc/src/main/java/dev/onvoid/webrtc/client/PeerImpl.java
Original file line number Diff line number Diff line change
Expand Up @@ -290,6 +290,27 @@ public Sender addTrack(LocalTrack track, String... streamIds) {
return addSender(local, ids, Collections.emptyList(), new CompletableFuture<>());
}

@Override
public Sender addTrack(LocalTrack track, Simulcast simulcast, String... streamIds) {
LocalTrackImpl local = checkTrack(track);
requireNonNull(simulcast, "simulcast");

if (local.kind() != MediaKind.VIDEO) {
throw new IllegalArgumentException("Simulcast takes a video track, got " + local.kind());
}

List<String> ids = new ArrayList<>();

for (String id : streamIds) {
ids.add(requireNonNull(id, "streamId"));
}
if (ids.isEmpty()) {
ids.add(defaultStreamId);
}

return addSender(local, ids, simulcast.encodings(), new CompletableFuture<>());
}

/**
* Adds a track to the stream of the tracks added without stream ids, sent
* in the given encodings, as simulcast (RFC 8853) does with one for each
Expand All @@ -308,7 +329,14 @@ CompletableFuture<Void> addTrack(LocalTrack track, List<RTCRtpEncodingParameters

private SenderImpl addSender(LocalTrackImpl local, List<String> ids, List<RTCRtpEncodingParameters> encodings,
CompletableFuture<Void> added) {
SenderImpl sender = new SenderImpl(this, local);
List<String> rids = new ArrayList<>();
for (RTCRtpEncodingParameters encoding : encodings) {
if (encoding.rid != null) {
rids.add(encoding.rid);
}
}

SenderImpl sender = new SenderImpl(this, local, rids);
senders.add(sender);

control.execute(() -> {
Expand Down
25 changes: 24 additions & 1 deletion webrtc/src/main/java/dev/onvoid/webrtc/client/Sender.java
Original file line number Diff line number Diff line change
Expand Up @@ -17,6 +17,7 @@
package dev.onvoid.webrtc.client;

import java.time.Duration;
import java.util.List;
import java.util.concurrent.CompletableFuture;
import java.util.function.Consumer;

Expand All @@ -42,14 +43,36 @@ public interface Sender {
CompletableFuture<Void> replaceTrack(LocalTrack track);

/**
* Changes how the track is encoded. Values left unset stay as they are.
* Changes how the track is encoded, in every layer alike if it is sent
* as {@link Simulcast}. Values left unset stay as they are.
*
* @param encoding the values to change.
*
* @return a future that completes once the values are applied.
*/
CompletableFuture<Void> configure(Encoding encoding);

/**
* Changes how one layer of a {@link Simulcast} track is encoded, such as
* pausing it with {@code active(false)} while no receiver wants it.
* Values left unset stay as they are.
*
* @param rid the rid of the layer, one of {@link #rids()}.
* @param encoding the values to change.
*
* @return a future that completes once the values are applied.
*
* @throws IllegalArgumentException if the sender has no layer of that
* rid.
*/
CompletableFuture<Void> configure(String rid, Encoding encoding);

/**
* @return the rids of the layers the track is sent in, in their order,
* or none if it is sent without simulcast.
*/
List<String> rids();

/**
* Sets the transform that every encoded frame of this sender passes
* through, after the encoder and before the packetizer, in place of the
Expand Down
77 changes: 63 additions & 14 deletions webrtc/src/main/java/dev/onvoid/webrtc/client/SenderImpl.java
Original file line number Diff line number Diff line change
Expand Up @@ -53,6 +53,8 @@ final class SenderImpl implements Sender {

private final MediaKind kind;

private final List<String> rids;

private volatile LocalTrackImpl track;

/**
Expand Down Expand Up @@ -85,10 +87,15 @@ final class SenderImpl implements Sender {
private final Handlers<String> toneHandlers;


SenderImpl(PeerImpl peer, LocalTrackImpl track) {
/**
* @param rids the rids of the simulcast layers the track is sent in, or
* none without simulcast.
*/
SenderImpl(PeerImpl peer, LocalTrackImpl track, List<String> rids) {
this.peer = peer;
this.track = track;
this.kind = track.kind();
this.rids = List.copyOf(rids);
this.toneHandlers = new Handlers<>(peer.rtc::reportError);
}

Expand Down Expand Up @@ -205,26 +212,68 @@ public CompletableFuture<Void> configure(Encoding encoding) {
RTCRtpSendParameters parameters = sender.getParameters();

for (RTCRtpEncodingParameters values : parameters.encodings) {
if (encoding.active != null) {
values.active = encoding.active;
}
if (encoding.maxBitrate != null) {
values.maxBitrate = encoding.maxBitrate;
}
if (kind == MediaKind.VIDEO) {
if (encoding.maxFrameRate != null) {
values.maxFramerate = encoding.maxFrameRate;
}
if (encoding.scaleDownBy != null) {
values.scaleResolutionDownBy = encoding.scaleDownBy;
}
apply(encoding, values);
}

sender.setParameters(parameters);
});
}

@Override
public CompletableFuture<Void> configure(String rid, Encoding encoding) {
if (rid == null) {
throw new NullPointerException("rid");
}
if (encoding == null) {
throw new NullPointerException("encoding");
}
if (!rids.contains(rid)) {
throw new IllegalArgumentException(rids.isEmpty()
? "The track is sent without simulcast, so it has no layer " + rid
: "The track is sent in the layers " + rids + ", not in " + rid);
}

return run(() -> {
RTCRtpSendParameters parameters = sender.getParameters();
boolean found = false;

for (RTCRtpEncodingParameters values : parameters.encodings) {
if (rid.equals(values.rid)) {
apply(encoding, values);
found = true;
}
}

if (!found) {
throw new WebRtcException("WebRTC reports no layer " + rid);
}

sender.setParameters(parameters);
});
}

@Override
public List<String> rids() {
return rids;
}

private void apply(Encoding encoding, RTCRtpEncodingParameters values) {
if (encoding.active != null) {
values.active = encoding.active;
}
if (encoding.maxBitrate != null) {
values.maxBitrate = encoding.maxBitrate;
}
if (kind == MediaKind.VIDEO) {
if (encoding.maxFrameRate != null) {
values.maxFramerate = encoding.maxFrameRate;
}
if (encoding.scaleDownBy != null) {
values.scaleResolutionDownBy = encoding.scaleDownBy;
}
}
}

@Override
public CompletableFuture<Void> transform(RTCEncodedFrameTransformer transform) {
return run(() -> sender.setTransform(peer.guard(transform, "a sender of " + peer.name)));
Expand Down
Loading
Loading