Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 3 additions & 1 deletion docs/guide/client-api-design.md
Original file line number Diff line number Diff line change
Expand Up @@ -140,7 +140,7 @@ and it hands on to the event executor what the application should hear.

| Code | Runs on | May it block? |
|---|---|---|
| Calls into the API | the caller's thread, any thread | They don't block, with these exceptions. `createPeer` waits for the connection, so that an invalid configuration fails there. The `LocalMedia` methods that make a track and `LocalTrack.setEnabled` wait briefly for a WebRTC thread, since the factory and the tracks are proxies. `camera()` and `screen()` also open their capture on the caller's thread. The first `devices()`, `screens()` and `windows()` ask the system there. `WebRtc.close()` waits up to `closeTimeout`, and a `join()` the caller writes waits as written. |
| Calls into the API | the caller's thread, any thread | They don't block, with these exceptions. `createPeer` waits for the connection, so that an invalid configuration fails there. The `LocalMedia` methods that make a track and `LocalTrack.setEnabled` and `LocalVideo.setContentHint` wait briefly for a WebRTC thread, since the factory and the tracks are proxies. `camera()` and `screen()` also open their capture on the caller's thread. The first `devices()`, `screens()` and `windows()` ask the system there. `WebRtc.close()` waits up to `closeTimeout`, and a `join()` the caller writes waits as written. |
| Event handlers, future callbacks, `Signaling.send` | the peer's event thread, or the configured `eventExecutor` | Yes, but it only delays that peer's events. It must not `join()` a future of this API: a direct `join()`/`get()` throws `BlockingCallException`. |
| `LocalTrack.onEnded`, `LocalAudio.onVoiceActivity`, `Devices.onChange`, futures of local tracks and devices | the event thread of the runtime's local media, or the configured `eventExecutor` | Yes, but it delays the other events of local media. The rule against `join()` holds here too. |
| Frame handlers (`onFrame`, `onAudio`) | a thread of each subscription, from the frame pool | Better not. A slow video handler gets the newest frame next, and the frames in between are dropped and released. A slow audio handler loses the oldest audio beyond its buffer. |
Expand Down Expand Up @@ -549,6 +549,8 @@ public interface LocalTrack extends AutoCloseable {
public interface LocalVideo extends LocalTrack {

Subscription onFrame(Consumer<VideoFrame> handler); // local preview
VideoContentHint contentHint();
void setContentHint(VideoContentHint hint); // motion, detail or text, also while sending
}

public interface LocalCamera extends LocalVideo {
Expand Down
20 changes: 19 additions & 1 deletion docs/guide/client-api.md
Original file line number Diff line number Diff line change
Expand Up @@ -290,7 +290,7 @@ Every method of the client API may be called from any thread, including from ins

| Code | Runs on | May it block? |
|---|---|---|
| Your calls into the API | your thread | They don't block, with these exceptions. `WebRtc.createPeer`, the `LocalMedia` methods that make a track and `LocalTrack.setEnabled` wait briefly for WebRTC. `camera()` and `screen()` also open their capture on your thread, which can take longer, so a UI thread should leave them to another thread. The first `media.devices()`, `screens()` and `windows()` ask the system on your thread. `WebRtc.close()` waits up to the close timeout. |
| Your calls into the API | your thread | They don't block, with these exceptions. `WebRtc.createPeer`, the `LocalMedia` methods that make a track and `LocalTrack.setEnabled` and `LocalVideo.setContentHint` wait briefly for WebRTC. `camera()` and `screen()` also open their capture on your thread, which can take longer, so a UI thread should leave them to another thread. The first `media.devices()`, `screens()` and `windows()` ask the system on your thread. `WebRtc.close()` waits up to the close timeout. |
| Peer, track and channel handlers, future callbacks, `Signaling.send` | the peer's event thread | Yes, but that delays only the events of this peer. They must not wait for a future of this API, see below. |
| `onFrame` and `onAudio` handlers | a thread of each subscription | Better not. A slow video handler gets the newest frame next, and the frames in between are dropped; a slow audio handler loses the oldest audio beyond its buffer. |
| [Encoded transforms](#encoded-transforms) | a thread of each sender and receiver | Better not. A slow transform delays only its own frames, and one that falls several seconds behind has frames dropped. |
Expand Down Expand Up @@ -483,6 +483,24 @@ Devices devices = media.devices();
LocalVideo screen = media.screen(ScreenOptions.screen(devices.screens().get(0)).frameRate(15).build());
```

A content hint tells the senders of a video track what to keep when the bandwidth or the CPU runs short. It can be changed at any time, also while the track is sent, without renegotiating:

```java
screen.setContentHint(VideoContentHint.TEXT); // source code: keep the resolution
peer.addTrack(screen);

screen.setContentHint(VideoContentHint.MOTION); // a video starts in the shared window: keep the frame rate
```

| Hint | Suits | Gives up first |
|---|---|---|
| `DEFAULT` | Anything | What the source suggests: the frame rate of screens, the resolution of other video |
| `MOTION` | Cameras, videos, animations | Resolution |
| `DETAIL` | Drawings, slides, photos | Frame rate |
| `TEXT` | Source code, terminals, tables | Frame rate; WebRTC treats it as `DETAIL` for now |

The hint belongs to the track, so it applies on every peer the track is sent on, and a camera keeps it when it switches. Screens keep their resolution by default already, so the hint pays off where the content changes, or for a camera that shows a whiteboard. No hint keeps text readable at any bitrate.

Tracks added without stream ids share one stream per peer, so that a browser plays their audio and video in sync. Pass ids to `addTrack(track, streamIds...)` to group them yourself.

One track may be sent on any number of peers. Opening a camera that is already open shares its capture when the options are the same, and fails with a `MediaException` when they differ.
Expand Down
13 changes: 13 additions & 0 deletions docs/guide/media/constraints.md
Original file line number Diff line number Diff line change
Expand Up @@ -142,6 +142,19 @@ sender.setParameters(parameters);

If unset, WebRTC keeps the resolution of screen shares and of tracks hinted as detailed or text, and the frame rate of other video.

### Content Hint

A content hint on the track says what it shows, and so decides the same for every sender of the track that has no `degradationPreference` of its own. It can be changed while the track is sent, without renegotiation, as when a video starts in a shared window:

```java
videoTrack.setContentHint(VideoContentHint.TEXT); // keep the resolution

// Later, a video plays in the same window.
videoTrack.setContentHint(VideoContentHint.MOTION); // keep the frame rate
```

`MOTION` keeps the frame rate, `DETAIL` and `TEXT` keep the resolution, and `DEFAULT` leaves it to the source, which keeps the resolution of screen shares. `DETAIL` and `TEXT` also put the encoder into its screen content mode, which VP8, VP9, H.264 (OpenH264) and AV1 code differently; WebRTC treats `TEXT` as `DETAIL` for now. The senders have taken the hint once `setContentHint` returns. It cannot be set from within the callback of a video sink, where it throws an `IllegalStateException`.

## Simulcast

With simulcast, a video sender encodes the same video several times, at different resolutions, so that a server can forward to each receiver the one its connection can take. The encodings are given when the transceiver is created, each with an RTP stream ID (`rid`):
Expand Down
5 changes: 5 additions & 0 deletions docs/guide/video/desktop-capture.md
Original file line number Diff line number Diff line change
Expand Up @@ -83,8 +83,13 @@ PeerConnectionFactory factory = new PeerConnectionFactory();

// Create a video track with the desktop source
VideoTrack videoTrack = factory.createVideoTrack("video0", videoSource);

// Keep the resolution, so that text stays readable, also when the bandwidth runs short.
videoTrack.setContentHint(VideoContentHint.TEXT);
```

The [content hint](/guide/media/constraints#content-hint) can be changed at any time, also while the track is sent.

### Adding the Track to a Peer Connection

Add the video track to your peer connection:
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -31,6 +31,7 @@
import dev.onvoid.webrtc.client.SignalingPair;
import dev.onvoid.webrtc.client.VideoStats;
import dev.onvoid.webrtc.client.WebRtc;
import dev.onvoid.webrtc.media.video.VideoContentHint;
import dev.onvoid.webrtc.media.video.desktop.DesktopSource;

/**
Expand Down Expand Up @@ -95,6 +96,10 @@ public static void main(String[] args) throws Exception {
// Frames larger than 1920x1080 are scaled down, keeping their aspect ratio.
LocalVideo screen = rtc.media().screen(options.frameRate(30).maxSize(1920, 1080).build());

// Keeps text readable when the bandwidth runs short, lowering the frame rate instead.
// Switch to MOTION while a video plays on the screen, the peers go on sending.
screen.setContentHint(VideoContentHint.TEXT);

System.out.println("Sharing " + describe(source));

// The two ends of a loopback signaling hand each message to the other peer.
Expand Down
16 changes: 16 additions & 0 deletions webrtc-jni/src/main/cpp/include/JNI_VideoTrack.h

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

30 changes: 29 additions & 1 deletion webrtc-jni/src/main/cpp/src/JNI_VideoTrack.cpp
Original file line number Diff line number Diff line change
Expand Up @@ -18,6 +18,7 @@
#include "JavaBoundary.h"
#include "NativeHandle.h"
#include "api/VideoTrackSink.h"
#include "JavaEnums.h"
#include "JavaNullPointerException.h"
#include "JavaUtils.h"

Expand Down Expand Up @@ -62,4 +63,31 @@ JNIEXPORT void JNICALL Java_dev_onvoid_webrtc_media_video_VideoTrack_removeSinkI
delete sink;
}
});
}
}

JNIEXPORT jobject JNICALL Java_dev_onvoid_webrtc_media_video_VideoTrack_getContentHint
(JNIEnv * env, jobject caller)
{
return jni::CallFromJava(env, [&]() -> jobject {
jni::NativeLease<webrtc::VideoTrackInterface> track(env, caller);
CHECK_HANDLEV(track, nullptr);

return jni::JavaEnums::toJava(env, track->content_hint()).release();
});
}

JNIEXPORT void JNICALL Java_dev_onvoid_webrtc_media_video_VideoTrack_setContentHintInternal
(JNIEnv * env, jobject caller, jobject jHint)
{
jni::CallFromJava(env, [&]() {
jni::NativeLease<webrtc::VideoTrackInterface> track(env, caller);
CHECK_HANDLE(track);

auto hint = jni::JavaEnums::toNative<webrtc::VideoTrackInterface::ContentHint>(env, jHint);

// The senders of the track reconfigure before this returns: the
// proxy runs it on the signaling thread, where the track notifies
// them, and each one waits for the worker thread.
track->set_content_hint(hint);
});
}
1 change: 1 addition & 0 deletions webrtc-jni/src/main/cpp/src/WebRTCContext.cpp
Original file line number Diff line number Diff line change
Expand Up @@ -89,6 +89,7 @@ namespace jni
JavaEnums::add<webrtc::DtxStatus>(env, PKG"RTCDtxStatus");
JavaEnums::add<webrtc::MediaSourceInterface::SourceState>(env, PKG_MEDIA"MediaSource$State");
JavaEnums::add<webrtc::MediaStreamTrackInterface::TrackState>(env, PKG_MEDIA"MediaStreamTrackState");
JavaEnums::add<webrtc::VideoTrackInterface::ContentHint>(env, PKG_MEDIA"video/VideoContentHint");
JavaEnums::add<webrtc::PeerConnectionInterface::BundlePolicy>(env, PKG"RTCBundlePolicy");
JavaEnums::add<webrtc::PeerConnectionInterface::IceConnectionState>(env, PKG"RTCIceConnectionState");
JavaEnums::add<webrtc::PeerConnectionInterface::IceGatheringState>(env, PKG"RTCIceGatheringState");
Expand Down
15 changes: 12 additions & 3 deletions webrtc/src/main/java/dev/onvoid/webrtc/client/LocalTrackImpl.java
Original file line number Diff line number Diff line change
Expand Up @@ -154,8 +154,17 @@ MediaStreamTrack nativeTrack() {
}

/**
* Puts another native track in place of the current one, keeping whether
* it is enabled.
* Gives a native track that is to replace the current one the settings
* of this track, before any sender sends it. Called under this object's
* lock.
*/
void adopt(MediaStreamTrack next) {
next.setEnabled(enabled);
}

/**
* Puts another native track in place of the current one, keeping its
* settings, such as whether it is enabled.
*
* @return the track and release replaced, and the senders that send it now;
* null once the track ended.
Expand All @@ -167,7 +176,7 @@ synchronized Replaced replaceNative(MediaStreamTrack next, Runnable nextRelease)

Replaced replaced = new Replaced(track, releaseSource, new ArrayList<>(senders));

next.setEnabled(enabled);
adopt(next);
track = next;
releaseSource = nextRelease;

Expand Down
25 changes: 25 additions & 0 deletions webrtc/src/main/java/dev/onvoid/webrtc/client/LocalVideo.java
Original file line number Diff line number Diff line change
Expand Up @@ -18,6 +18,7 @@

import java.util.function.Consumer;

import dev.onvoid.webrtc.media.video.VideoContentHint;
import dev.onvoid.webrtc.media.video.VideoFrame;

/**
Expand All @@ -37,4 +38,28 @@ public interface LocalVideo extends LocalTrack {
*/
Subscription onFrame(Consumer<VideoFrame> handler);

/**
* @return what the track is hinted to show, {@link
* VideoContentHint#DEFAULT} unless set.
*/
VideoContentHint contentHint();

/**
* Hints what the track shows, which decides what its senders keep when
* the network or the CPU cannot keep up: the frame rate for {@link
* VideoContentHint#MOTION}, the resolution for {@link
* VideoContentHint#DETAIL} and {@link VideoContentHint#TEXT}. Can be
* changed while the track is sent, without renegotiating, as when a video
* starts playing in a shared window.
* <p>
* The hint belongs to the track: it applies on every peer the track is
* sent on, and stays when a camera switches. Every sender has taken the
* hint once this returns; a change may make a sender reconfigure its
* encoder. Waits briefly for a WebRTC thread. Does nothing once the
* track ended.
*
* @param hint what the track shows.
*/
void setContentHint(VideoContentHint hint);

}
29 changes: 29 additions & 0 deletions webrtc/src/main/java/dev/onvoid/webrtc/client/LocalVideoImpl.java
Original file line number Diff line number Diff line change
Expand Up @@ -22,6 +22,8 @@
import java.util.List;
import java.util.function.Consumer;

import dev.onvoid.webrtc.media.MediaStreamTrack;
import dev.onvoid.webrtc.media.video.VideoContentHint;
import dev.onvoid.webrtc.media.video.VideoFrame;
import dev.onvoid.webrtc.media.video.VideoTrack;

Expand All @@ -34,6 +36,9 @@ class LocalVideoImpl extends LocalTrackImpl implements LocalVideo {

private final List<FrameDispatcher> dispatchers = new ArrayList<>();

/** Written under this, kept for the native track a camera switch makes. */
private volatile VideoContentHint contentHint = VideoContentHint.DEFAULT;


LocalVideoImpl(LocalMediaImpl media, VideoTrack track, Runnable releaseSource) {
super(media, track, MediaKind.VIDEO, releaseSource);
Expand Down Expand Up @@ -67,6 +72,30 @@ public Subscription onFrame(Consumer<VideoFrame> handler) {
};
}

@Override
public VideoContentHint contentHint() {
return contentHint;
}

@Override
public synchronized void setContentHint(VideoContentHint hint) {
requireNonNull(hint, "hint");

if (!isLive()) {
return;
}

videoTrack().setContentHint(hint);
contentHint = hint;
}

@Override
void adopt(MediaStreamTrack next) {
super.adopt(next);

((VideoTrack) next).setContentHint(contentHint);
}

@Override
void beforeRelease() {
List<FrameDispatcher> attached;
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -208,7 +208,12 @@ public static void awaitDeferred() throws InterruptedException {
done.await();
}

private static boolean inCallback() {
/**
* @return true while the current thread hands media to a sink, of any
* track, and holds the lock of WebRTC's broadcaster that adding a sink
* takes.
*/
public static boolean inCallback() {
return CALLBACK_DEPTH.get()[0] > 0;
}

Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,60 @@
/*
* Copyright 2026 Alex Andres
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/

package dev.onvoid.webrtc.media.video;

/**
* What a video track shows, which decides what its senders keep when the
* network or the CPU cannot keep up: the frame rate or the resolution. See
* <a href="https://w3c.github.io/mst-content-hint/">MediaStreamTrack Content
* Hints</a>.
* <p>
* A {@link dev.onvoid.webrtc.RTCRtpSendParameters#degradationPreference
* degradation preference} set on a sender takes precedence over the hint in
* that choice.
*
* @author Alex Andres
*
* @see VideoTrack#setContentHint(VideoContentHint)
*/
public enum VideoContentHint {

/**
* No hint: the source decides. Screen shares keep the resolution, other
* video the frame rate.
*/
DEFAULT,

/**
* Motion, such as a camera, a video or an animation: keep the frame rate,
* and lower the resolution where necessary.
*/
MOTION,

/**
* Detail, such as drawings, slides or photos: keep the resolution, and
* lower the frame rate where necessary.
*/
DETAIL,

/**
* Text, such as source code, a terminal or tables: keep the resolution,
* as with {@link #DETAIL}, which WebRTC treats it as for now, with every
* codec. No hint keeps text readable at any bitrate.
*/
TEXT

}
Loading
Loading