From 2b76bf3a15feb8200b74c83ca6f44a661a6210c6 Mon Sep 17 00:00:00 2001 From: Alex Andres Date: Thu, 12 Nov 2026 19:19:16 +0100 Subject: [PATCH 01/37] docs: write the WHIP and WHEP page in shorter sentences Same content: long sentences split, sequences as numbered lists, rules as bullet lists. --- docs/guide/client/whip-whep.md | 163 ++++++++++++++++++++++++++------- 1 file changed, 128 insertions(+), 35 deletions(-) diff --git a/docs/guide/client/whip-whep.md b/docs/guide/client/whip-whep.md index e74cac19..fa8986f8 100644 --- a/docs/guide/client/whip-whep.md +++ b/docs/guide/client/whip-whep.md @@ -1,6 +1,6 @@ # WHIP and WHEP -Media servers and streaming services take streams with the WebRTC HTTP Ingestion Protocol ([WHIP](https://www.rfc-editor.org/rfc/rfc9725)) and hand them out with its egress counterpart, WHEP. Both exchange one offer and one answer over HTTP. `Whip` publishes tracks, and `Whep` receives a stream: +Media servers and streaming services take streams over WHIP, the WebRTC HTTP Ingestion Protocol ([RFC 9725](https://www.rfc-editor.org/rfc/rfc9725)), and hand them out over its counterpart WHEP. Both exchange one offer and one answer over HTTP. `Whip` publishes tracks, and `Whep` receives a stream: ```java Whip.Session live = Whip.to(URI.create("https://example.com/live/whip")) @@ -19,13 +19,21 @@ viewer.peer().onTrack(track -> System.out.println("Receiving " + track.kind())); ## Connecting -By default, the offer goes to the endpoint at once, before any candidate is gathered, and the future completes once the peer is connected. If the endpoint refuses the offer, or the peer does not connect within the timeout (15 seconds by default), the future fails, and the session is deleted on the server again. Cancelling the future, say once `get` gave up waiting, ends the session the same way. `onTrack` hands over tracks that already arrived, so a handler added once the future completed misses none. Retries and counter-offers are described under [Details](#retries-and-counter-offers). +By default, `publish` and `play` send the offer to the endpoint at once, before any candidate is gathered. Their future completes once the peer is connected. + +The future fails if the endpoint refuses the offer, or if the peer does not connect within the timeout (15 seconds by default). The session is then deleted on the server again. Cancelling the future, for example after `get` gave up waiting, ends the session the same way. + +`onTrack` also hands over tracks that arrived before the handler was added. A handler added after the future completed misses none. + +An endpoint may ask for the offer again later, and a WHEP endpoint may answer with an offer of its own. [Details](#retries-and-counter-offers) describes both. ## Candidates -With the default `Gathering.TRICKLE`, only once the endpoint answered does the peer gather its candidates, with the STUN and TURN servers the endpoint names in `Link` headers with `rel="ice-server"`, unless `iceServer(...)` or `turnServer(...)` names servers of its own, which take precedence. The candidates follow the offer by `PATCH` to the session, as `application/trickle-ice-sdpfrag`, a few at a time and with the session's `ETag` as `If-Match`, as RFC 9725 describes. +By default (`Gathering.TRICKLE`), the peer gathers its candidates only after the endpoint has answered. It then sends them to the session a few at a time, by `PATCH`. -For an endpoint without Trickle ICE, use `.gathering(Gathering.COMPLETE)` to include gathered candidates in the initial POST. It waits at most three seconds for gathering, within the overall session timeout, then sends the candidates available so far; any later candidates may still follow by PATCH. Configure STUN/TURN servers locally in this mode, since the endpoint's Link headers arrive after gathering. For example: +The peer uses the STUN and TURN servers that the endpoint names in its `Link` headers (`rel="ice-server"`). If you set servers with `iceServer(...)` or `turnServer(...)`, those are used instead. + +An endpoint without Trickle ICE needs the candidates in the first request. `gathering(Gathering.COMPLETE)` puts them there: ```java Whep.Session viewer = Whep.from(endpoint) @@ -35,17 +43,28 @@ Whep.Session viewer = Whep.from(endpoint) .get(15, TimeUnit.SECONDS); ``` -Both builders support these methods. `turnServer` is separate from `iceServer(String...)`, so passing three URLs to `iceServer` still means three URLs. The gathering setting controls the initial POST only; ICE restarts continue to use PATCH and trickled candidates. +- The peer waits up to three seconds for gathering, within the session's timeout, then sends the candidates it has. Later ones may still follow by `PATCH`. +- Set the STUN and TURN servers yourself in this mode. The endpoint's `Link` headers arrive only after gathering. +- The gathering mode affects only the first request. ICE restarts always use `PATCH` and trickled candidates. + +Both builders have `turnServer` and `gathering`. `turnServer` takes one TURN URL with its user name and password. `iceServer(String...)` takes URLs only, so three arguments are three URLs. -How the session treats an endpoint that refuses candidates is described under [Details](#trickle-ice-responses). +How the session treats an endpoint that refuses candidates is described under [Details](#trickle-ice). ## Codecs -By default, the offer holds every codec WebRTC has, in WebRTC's order, and the endpoint picks; one that follows the offer's order picks VP8 for video. Endpoints that pass the stream on unchanged, to HLS or a recording, often take only H.264 and Opus. `videoCodecs(...)` and `audioCodecs(...)` offer only the codecs named, in that order of preference, by name (`H264`, `VP8`, `VP9`, `AV1`, `opus`, `G722`, ...) or MIME type (`video/H264`), whatever the case. Retransmission, redundancy, FEC, comfort noise and DTMF go along. For WHEP, they decide what the endpoint sends, also in the answer to a counter-offer. A name WebRTC has no codec of fails the session with a `ConfigurationException` before the offer goes out. +By default, the offer holds every codec WebRTC has, in WebRTC's order, and the endpoint picks. An endpoint that follows the offer's order picks VP8 for video. + +Endpoints that pass the stream on unchanged, for example to HLS or into a recording, often take only H.264 and Opus. `videoCodecs(...)` and `audioCodecs(...)` offer only the codecs you name, in your order of preference: + +- Name a codec by its name (`H264`, `VP8`, `VP9`, `AV1`, `opus`, `G722`, ...) or by its MIME type (`video/H264`). Case does not matter. +- Retransmission, redundancy, FEC, comfort noise and DTMF stay in the offer. +- For WHEP, the codecs decide what the endpoint sends. This also holds for the answer to a counter-offer. +- A name WebRTC has no codec for fails the session with a `ConfigurationException`, before the offer goes out. ## Endpoints That Require Video -Some endpoints, Amazon IVS among them, refuse a WHIP offer without video, even for audio alone. `alwaysOfferVideo()` then adds video that only sends, in the stream of the tracks, without a track, so that nothing is sent on it; with a video track, it changes nothing: +Some endpoints, Amazon IVS among them, refuse a WHIP offer without video, even when only audio is sent. `alwaysOfferVideo()` then adds send-only video to the offer, in the stream of the tracks. It has no track, so nothing is sent on it. With a video track, `alwaysOfferVideo()` changes nothing: ```java Whip.Session live = Whip.to(URI.create("https://example.com/live/whip")) @@ -60,7 +79,9 @@ Whip.Session live = Whip.to(URI.create("https://example.com/live/whip")) ## Simulcast -`simulcast()` sends the video track of a WHIP session in three layers, as simulcast (RFC 8853), in full, half and quarter resolution, with the rids `f`, `h` and `q`. The offer negotiates them, as RFC 9725 asks, and an endpoint that forwards the stream can then give each viewer the layer its bandwidth takes. `simulcast(Encoding...)` takes two or three layers of its own, the most WebRTC sends, from the largest down, with their resolution factor, bitrate and frame rate. An endpoint that takes no simulcast leaves it out of its answer and gets the first layer only, which is why the largest comes first. Simulcast needs a video track; `Sender.configure` on its sender changes every layer alike. +`simulcast()` sends the video track of a WHIP session in three layers, as simulcast (RFC 8853): full, half and quarter resolution, with the rids `f`, `h` and `q`. An endpoint that forwards the stream can then give each viewer the layer its bandwidth allows. The offer negotiates the layers, as RFC 9725 asks. + +`simulcast(Encoding...)` takes two or three layers of your own, each with its resolution factor, bitrate and frame rate. WebRTC sends at most three. List the largest first: an endpoint without simulcast leaves it out of its answer and gets only the first layer. ```java Whip.Session live = Whip.to(URI.create("https://example.com/live/whip")) @@ -74,9 +95,18 @@ Whip.Session live = Whip.to(URI.create("https://example.com/live/whip")) .get(15, TimeUnit.SECONDS); ``` +Simulcast needs a video track. `Sender.configure` on its sender changes every layer alike. + ## Errors -If an endpoint answers with an error, the message of the failure names the status, and what the endpoint says of it: the title and detail of problem details (RFC 9457, `application/problem+json`), as in `status 403: Forbidden: The stream key has expired`, or else the start of the body. The failure is an `HttpStatusException`, whose `status()`, `title()`, `detail()` and `problemType()` give the same to a program; for a deletion or for candidates, it is the cause of the failure reported. As RFC 9725 asks, read no specific cause into a status beyond its class, since endpoints use them differently: +If the endpoint answers with an error status, the session fails with an `HttpStatusException`. Its message names the status and what the endpoint says about it, for example `status 403: Forbidden: The stream key has expired`: + +- With problem details (RFC 9457, `application/problem+json`), the message holds their title and detail. +- Without them, it holds the start of the body. + +`status()`, `title()`, `detail()` and `problemType()` give the same to a program. If a deletion or a candidate request fails, the reported failure has the `HttpStatusException` as its cause. + +As RFC 9725 asks, read no specific cause into a status beyond its class, since endpoints use statuses differently: ```java try { @@ -92,17 +122,32 @@ catch (ExecutionException e) { ## ICE Restarts -When the connection fails, or stays disconnected for two seconds, the session restarts ICE: the peer gathers new candidates, and only the new ICE username fragment and password go to the session, by `PATCH` with `If-Match: *`, while the media go on as negotiated. The session's answer, with status 200 and Content-Type `application/trickle-ice-sdpfrag`, brings its own ICE information and a new `ETag`. Its candidate set replaces the old one completely, including when it is empty. Three restarts in a row that do not bring the connection back, or an endpoint that takes no restarts, end the attempts, which the error handler hears of. `session.restartIce()` restarts ICE at once, say after the network changed; a restart that fails leaves the ICE session before as it was. Restarts also work after a WHEP counter-offer; the player keeps its answerer role and updates only the ICE information. +The session restarts ICE when the connection fails, or when it stays disconnected for two seconds. The media go on as negotiated; only the ICE information changes: + +1. The peer gathers new candidates. +2. Its new ICE username fragment and password go to the session by `PATCH`, with `If-Match: *`. +3. The session answers with status 200 and Content-Type `application/trickle-ice-sdpfrag`. The answer brings the session's ICE information and a new `ETag`. +4. The candidates of the answer replace the old ones completely, even when there are none. + +The attempts end after three restarts in a row that do not bring the connection back, or when the endpoint takes no restarts. The error handler hears of it. + +`session.restartIce()` restarts ICE at once, for example after the network changed. A restart that fails leaves the previous ICE session as it was. Restarts also work after a WHEP counter-offer: the player stays the answering side and updates only the ICE information. ## Authentication -Servers that ask for a user and password rather than a token, such as MediaMTX, take them by HTTP Basic authentication (RFC 7617): `basicAuth("user", "password")` sends `Authorization: Basic` with both, in UTF-8, on every request of the session, without waiting to be asked. The user must not contain a colon. Of `bearerToken` and `basicAuth`, the one called last counts. MediaMTX also takes them as a token, the way OBS Studio sends them, `bearerToken("user:password")`. Other schemes, such as Digest, are not supported. +`bearerToken(...)` sends a token as `Authorization: Bearer`, for many services the stream key. Servers that want a user and password instead, such as MediaMTX, take them by HTTP Basic authentication (RFC 7617): -How requests reach the endpoint, over `https`, redirects and proxies, is described under [Details](#https-and-redirects). +- `basicAuth("user", "password")` sends `Authorization: Basic` with both, in UTF-8, on every request of the session, without waiting to be asked. +- The user must not contain a colon. +- Of `bearerToken` and `basicAuth`, the one called last counts. +- MediaMTX also takes them as a token, the way OBS Studio sends them: `bearerToken("user:password")`. +- Other schemes, such as Digest, are not supported. + +Endpoints must be `https` URLs, except for a media server on the same machine. The rules for `https`, redirects and proxies are under [Details](#https-and-redirects). ## Detecting a WHEP Endpoint -`Whep.from(url).detect(rtc)` asks a URL with `HEAD` whether it is a WHEP endpoint, as a player may before it plays a URL it was given (draft-ietf-wish-whep-05, 4.2). Its future holds `true` when the endpoint answers 200 with the Content-Type `application/sdp` and a `Link` to the profile `urn:ietf:params:whep`; the Content-Type alone does not count. It holds `false` for any other answer, such as that of an endpoint that takes no `HEAD`, and fails when the request does. The request carries the credentials of the builder, takes its timeout at most, and follows the redirects the offer follows: +A player that was given a URL may first ask whether it is a WHEP endpoint (draft-ietf-wish-whep-05, 4.2). `Whep.from(url).detect(rtc)` asks with a `HEAD` request: ```java if (Whep.from(url).detect(rtc).get(15, TimeUnit.SECONDS)) { @@ -110,67 +155,115 @@ if (Whep.from(url).detect(rtc).get(15, TimeUnit.SECONDS)) { } ``` +- The future holds `true` if the endpoint answers 200 with Content-Type `application/sdp` and a `Link` to the profile `urn:ietf:params:whep`. The Content-Type alone is not enough. +- It holds `false` for any other answer, for example from an endpoint that takes no `HEAD`. +- It fails if the request itself fails. + +The request carries the builder's credentials, takes at most the builder's timeout, and follows the same redirects as the offer. + ## Closing a Session -`session.close()` deletes the session on the server, which frees what the server holds for it, and closes the peer. Both start at once, so a slow server does not keep the peer open, and the deletion takes at most the session's timeout. The tracks a WHIP session published stay open. `session.peer()` is a regular peer, for stats and anything else. A deletion the server refuses or that fails goes to the runtime's error handler, and also fails the future of `closeAsync()`, which completes once both are done; a peer that did not close fails it too. +`session.close()` deletes the session on the server, which frees what the server holds for it, and closes the peer. Both start at once, so a slow server does not keep the peer open. The deletion takes at most the session's timeout. + +The tracks a WHIP session published stay open. `session.peer()` is a regular peer, for statistics and anything else. + +`closeAsync()` returns a future that completes once the deletion and the peer's close are both done. It fails if the server refuses the deletion, if the deletion fails, or if the peer does not close. A deletion that the server refuses or that fails also goes to the runtime's error handler. ## Builder Methods | Builder method | What it sets | |---|---| | `bearerToken(String)` | The token sent as `Authorization: Bearer`; for many services, the stream key. | -| `basicAuth(String, String)` | A user and password sent as `Authorization: Basic`, as MediaMTX takes them; replaces a bearer token, and the other way round. | -| `iceServer(String...)` | A STUN or TURN server without credentials, overriding endpoint-provided servers. | -| `turnServer(String, String, String)` | A TURN URL, username and password, overriding endpoint-provided servers. | -| `gathering(Gathering)` | When the initial offer is sent: `TRICKLE` by default, or `COMPLETE` to gather first. | -| `track(LocalTrack)` | A track to publish, WHIP only. The tracks share one stream, which holds one audio and one video track at most. | +| `basicAuth(String, String)` | A user and password sent as `Authorization: Basic`. Replaces a bearer token, and the other way round. | +| `iceServer(String...)` | A STUN or TURN server without credentials, in place of the endpoint's servers. | +| `turnServer(String, String, String)` | A TURN URL, user name and password, in place of the endpoint's servers. | +| `gathering(Gathering)` | When the first offer is sent: `TRICKLE` by default, or `COMPLETE` to gather first. | +| `track(LocalTrack)` | A track to publish, WHIP only. The tracks share one stream, of one audio and one video track at most. | | `receive(MediaKind...)` | What to receive, WHEP only; video and audio by default. | | `videoCodecs(String...)`, `audioCodecs(String...)` | The only codecs to negotiate, in order of preference; every codec WebRTC has by default. | -| `alwaysOfferVideo()` | Video in the offer even without a video track, WHIP only, for endpoints that refuse an offer without video. | -| `simulcast()`, `simulcast(Encoding...)` | The video track in layers, as simulcast, WHIP only: three by default (`f`, `h`, `q`), or two or three given, the largest first. | +| `alwaysOfferVideo()` | Video in the offer even without a video track, WHIP only. | +| `simulcast()`, `simulcast(Encoding...)` | The video track in simulcast layers, WHIP only: three by default (`f`, `h`, `q`), or two or three given, the largest first. | | `sendTransform(RTCEncodedFrameTransformer)` | The [encoded transform](/guide/client/transforms) of every frame sent, WHIP only, in place before the offer goes out. | | `e2ee(KeyProvider)` | [End-to-end encryption](/guide/advanced/end-to-end-encryption) of every frame sent (WHIP) or received (WHEP), in place of the transforms. | | `receiveTransform(RTCEncodedFrameTransformer)` | The [encoded transform](/guide/client/transforms) of every frame received, WHEP only, in place before the first frame, also for a counter-offer. | -| `timeout(Duration)` | How long the session may take to be connected, and its deletion to be answered. | +| `timeout(Duration)` | How long the session may take to connect, and its deletion to be answered. | ## Details ### Offers -As RFC 9725 asks, the peer bundles all media on one transport (`max-bundle`), the descriptions it sends mark each m-line `rtcp-mux-only`, and a WHIP offer only sends (`sendonly`). +As RFC 9725 asks: + +- the peer bundles all media on one transport (`max-bundle`), +- the descriptions it sends mark each m-line `rtcp-mux-only`, +- a WHIP offer only sends (`sendonly`). ### Retries and Counter-Offers -An endpoint that is busy, with status 503, or a WHEP endpoint whose stream has yet to start, with status 409, gets the offer again after the wait its `Retry-After` names, at least a second, as long as that fits within the timeout; without `Retry-After` the session fails at once. +Two statuses make the session send its offer again: 503, when the endpoint is busy, and for WHEP 409, when the stream has yet to start. The session waits as long as the answer's `Retry-After` says, at least a second, as long as that fits within the timeout. Without `Retry-After`, the session fails at once. + +A WHEP endpoint may also reject the offer with status 406 and counter it with an offer of its own. `Whep` then: -A WHEP endpoint may also reject the offer and counter it with one of its own, with status 406: `Whep` then takes its offer back, answers the endpoint's, receiving only, and sends the answer by `PATCH` to the session the endpoint named, before the counter-offer's `valid-until`, 30 seconds by default. The answer rejects the audio or video of the counter-offer that `receive(...)` leaves out, and the session fails if the counter-offer holds none of what it asks for. Data channels of a counter-offer are accepted. +1. takes its own offer back, +2. answers the endpoint's offer, receiving only, +3. sends the answer by `PATCH` to the session the endpoint named, before the counter-offer's `valid-until` (30 seconds by default). -### Trickle ICE Responses +The answer rejects the audio or video of the counter-offer that `receive(...)` leaves out. If the counter-offer holds nothing of what `receive(...)` asks for, the session fails. Data channels of a counter-offer are accepted. -An endpoint that takes no such `PATCH`, with status 405, 415, 422 or 501, gets no more; it may still learn peer-reflexive candidates from incoming connectivity checks, but that does not cover every network topology. A 412 or 428 for the current ICE generation reaches the error handler and stops candidate PATCHes; responses to a generation replaced by a restart are ignored. +### Trickle ICE + +Candidates go to the session as `application/trickle-ice-sdpfrag`, with the session's `ETag` as `If-Match`, as RFC 9725 describes. + +An endpoint that refuses them with status 405, 415, 422 or 501 gets no more. It may still learn peer-reflexive candidates from incoming connectivity checks, but that does not cover every network topology. + +A 412 or 428 for the current ICE generation reaches the error handler and stops the candidate requests. Such answers for a generation that a restart replaced are ignored. ### Responses -A normal SDP answer must have status 201, a nonempty session `Location`, and Content-Type `application/sdp` (parameters are allowed). Invalid responses fail the session; a resource already named by the server is deleted even when its SDP response cannot be used. +An SDP answer must have status 201, a nonempty session `Location`, and Content-Type `application/sdp` (parameters are allowed). An invalid answer fails the session. A session the server already named is deleted, even when its SDP answer cannot be used. ### Timeouts -The timeout bounds the requests as a whole, not each read, so a server that sends its answer a byte at a time cannot stretch it. A request still out when the session times out or is cancelled is cut off at once; one still waiting for the head of its answer only from Java 16 on, and before that at its timeout. Once the server has named the session, in the head of its answer, it is deleted all the same; a session the server created for an answer that had not come yet is left for the server to time out. +The timeout bounds each request as a whole, not each read. A server that sends its answer a byte at a time cannot stretch it. + +When the session times out or is cancelled, requests still out are cut off at once. For a request still waiting for the head of its answer, that works only from Java 16 on; before that, the request ends at its timeout. + +Once the server has named the session in the head of its answer, the session is deleted all the same. A session the server created for an answer that has not arrived yet is left for the server to time out. ### URLs in Failures -Failures and reports name the URL of an endpoint or session by its scheme, host and port only, since the rest may be a secret: services take stream keys and signatures in the query, and some, such as Cloudflare Stream, in the path. A failure can go to a log as it is, apart from what the endpoint itself says in an error, which it passes on unchanged. +Failures and reports name the URL of an endpoint or session by its scheme, host and port only. The rest may be a secret: services take stream keys and signatures in the query, and some, such as Cloudflare Stream, in the path. + +A failure can therefore go to a log as it is. The exception is what the endpoint itself says in an error, which is passed on unchanged. ### HTTPS and Redirects -Endpoints have to be `https` URLs, as WHIP and WHEP ask, except on the machine the application runs on: there plain `http` will do, for a media server beside it, at `localhost`, an address in 127.0.0.0/8 or `::1`. A name that merely resolves to one of them does not count, since it may resolve elsewhere later. The same holds for the session an endpoint names and for where a redirect points. +Endpoints must be `https` URLs, as WHIP and WHEP ask. The same holds for the session an endpoint names and for redirect targets. + +The exception is a media server on the same machine: there plain `http` works, at `localhost`, an address in 127.0.0.0/8, or `::1`. A name that merely resolves to one of them does not count, since it may resolve elsewhere later. + +The credentials of a session, its bearer token or its Basic user and password, go to the endpoint. They also go, with the deletion, to the session the endpoint names in its `Location` header. That session may be on another host, but an `https` endpoint cannot name a plain `http` one. -The credentials of a session, its bearer token or its Basic user and password, go to the endpoint, and along with the deletion to the session the endpoint names in its `Location` header. That session may be on another host, but an `https` endpoint cannot name a plain `http` one. The offer follows redirects with status 307 and 308, as WHIP and WHEP ask of clients, such as those of a load balancer: up to five of them, all within the timeout, and the credentials go along. A redirect from `https` to plain `http` fails the session, and so does one with status 301, 302 or 303, which would turn the offer into a `GET`. The deletion follows no redirect; one fails it with its status. +Redirects: + +- The offer follows redirects with status 307 and 308, as WHIP and WHEP ask of clients, for example behind a load balancer. It follows up to five of them, all within the timeout, and the credentials go along. +- A redirect from `https` to plain `http` fails the session. +- So does a redirect with status 301, 302 or 303, which would turn the offer into a `GET`. +- The deletion follows no redirect. A redirect fails it with its status. ### Proxies -Requests go through Java's `HttpClient`, with the default `SSLContext` and the HTTP proxy Java is configured with, the first the default `ProxySelector` names: `-Dhttps.proxyHost` and `-Dhttps.proxyPort`, `-Dhttp.proxyHost`, or `-Djava.net.useSystemProxies=true`. `HttpClient` takes no SOCKS proxies. A proxy that asks for credentials gets Basic ones from the default `Authenticator`, but only for sessions without credentials of their own: `HttpClient` drops the Authorization header of a request once it answers for credentials itself. As in Java, Basic is not used for the tunnel of an `https` request unless `jdk.http.auth.tunneling.disabledSchemes` allows it. Other schemes, such as Digest, NTLM and Negotiate, are not supported. +Requests go through Java's `HttpClient`, with the default `SSLContext` and the HTTP proxy Java is configured with. That is the first proxy the default `ProxySelector` names, set by `-Dhttps.proxyHost` and `-Dhttps.proxyPort`, `-Dhttp.proxyHost`, or `-Djava.net.useSystemProxies=true`. `HttpClient` takes no SOCKS proxies. + +A proxy that asks for credentials gets Basic ones from the default `Authenticator`, but only for sessions without credentials of their own. Once `HttpClient` answers for credentials itself, it drops the Authorization header of the request. + +As elsewhere in Java, Basic is not used for the tunnel of an `https` request unless `jdk.http.auth.tunneling.disabledSchemes` allows it. Other schemes, such as Digest, NTLM and Negotiate, are not supported. ### Protocol Extensions -`session.extensions()` gives the protocol extensions the endpoint offers for the session (RFC 9725, 4.9): the URL of each by its relation type, such as `urn:ietf:params:whip:ext:example:server-sent-events` (`urn:ietf:params:whep:ext:...` for WHEP), as the endpoint named them in `Link` header fields of its answer, resolved against it. Of a relation type named twice, the first counts. Links of other relation types are ignored, as WHIP and WHEP ask, apart from `ice-server`. The session uses none of the extensions itself; an application uses those it knows, and the map is empty if the endpoint named none. +`session.extensions()` gives the protocol extensions the endpoint offers for the session (RFC 9725, 4.9), as the `Link` header fields of its answer name them. The map holds the URL of each extension by its relation type, such as `urn:ietf:params:whip:ext:example:server-sent-events` (`urn:ietf:params:whep:ext:...` for WHEP). Each URL is resolved against the URL that answered. + +- Of a relation type named twice, the first counts. +- Links of other relation types are ignored, as WHIP and WHEP ask, apart from `ice-server`. +- The session uses none of the extensions itself. An application uses those it knows. +- The map is empty if the endpoint named none. From aa38375b3ffaa5336041b4327913ba0086c915e9 Mon Sep 17 00:00:00 2001 From: Alex Andres Date: Thu, 12 Nov 2026 19:33:22 +0100 Subject: [PATCH 02/37] docs: write the media files page in shorter sentences Same content: platforms and measurements as tables, rules as lists, and the page names its client API counterpart. --- docs/guide/media/media-files.md | 204 +++++++++++++++++++++++--------- 1 file changed, 148 insertions(+), 56 deletions(-) diff --git a/docs/guide/media/media-files.md b/docs/guide/media/media-files.md index 8c3967fb..60783c55 100644 --- a/docs/guide/media/media-files.md +++ b/docs/guide/media/media-files.md @@ -1,37 +1,30 @@ # Media Files -This guide explains how to send a media file over a peer connection instead of a camera and a microphone, using the `webrtc-java-media` module. It covers: + -- Adding the media module to a project -- Sending a file with `MediaFileSource` -- Reading what a source contains with `MediaReader` and `MediaInfo` -- Controlling playback and following it with a listener -- Feeding your own media sources with `MediaPlayer` -- Playing a live stream from an IP camera or media server over RTSP +The `webrtc-java-media` module sends a media file over a peer connection, in place of a camera and a microphone. It also plays live streams from IP cameras and media servers over RTSP. -Sending a file is a common need: a test pattern instead of a webcam, a pre-recorded briefing, a video that has to reach several participants. Without help, an application has to bring its own decoder and push I420 frames into a `CustomVideoSource` itself. The media module removes that work by decoding with [FFmpeg](https://ffmpeg.org) inside the library. - -Decoding happens entirely in native code. Frames never travel through Java: the module hands decoded pictures straight to the native side of a `CustomVideoSource`, and 10 ms chunks to a `CustomAudioSource`. They are paced in real time and carry the presentation times of the file, so what a receiver gets keeps the timing of the media rather than the timing of a Java thread. +Sending a file is a common need: a test pattern instead of a webcam, a pre-recorded briefing, or a video that has to reach several participants. Without the module, an application has to bring its own decoder and push I420 frames into a `CustomVideoSource` itself. The module decodes with [FFmpeg](https://ffmpeg.org) inside the library instead. ## Adding the Module -The module is part of the normal build, and it builds FFmpeg and dav1d (the software decoder of AV1) from submodules pinned to release tags, so the submodules have to be present: +The module is part of the normal build. It builds FFmpeg and dav1d, the software decoder of AV1, from submodules pinned to release tags, so the submodules have to be present: ```shell git submodule update --init --depth 1 webrtc-media/third-party/ffmpeg webrtc-media/third-party/dav1d mvn install ``` -Building FFmpeg needs `make` and `nasm`, and building dav1d needs [Meson](https://mesonbuild.com) (`pip install meson`) and Ninja. On Windows `make` and `nasm` come from MSYS2: +Building FFmpeg needs `make` and `nasm`. Building dav1d needs [Meson](https://mesonbuild.com) (`pip install meson`) and Ninja. On Windows, `make` and `nasm` come from MSYS2: ```shell winget install MSYS2.MSYS2 C:\msys64\usr\bin\bash -lc "pacman -S --needed make nasm diffutils pkgconf" ``` -Maven still runs from an ordinary shell; the build enters MSYS2 and the Visual Studio environment on its own. The first build compiles FFmpeg and dav1d, which takes a while; later builds reuse the install directory. +Maven still runs from an ordinary shell: the build enters MSYS2 and the Visual Studio environment by itself. The first build compiles FFmpeg and dav1d, which takes a while. Later builds reuse the install directory. -Once installed, depend on it alongside `webrtc-java`. It takes two entries: one for the Java API, and one for the natives of the platform you are running on. +Once installed, depend on the module beside `webrtc-java`. It takes two entries: one for the Java API, and one for the natives of the platform you run on: ```xml @@ -47,15 +40,15 @@ Once installed, depend on it alongside `webrtc-java`. It takes two entries: one ``` -The classifier jar carries the module's native library together with the FFmpeg shared libraries it uses, so applications that do not use this module never download FFmpeg. Replace the classifier with the platform you are building for: `windows-x86_64`, `windows-aarch64`, `linux-x86_64`, `linux-aarch64`, `linux-aarch32`, `macos-x86_64` or `macos-aarch64`. +The classifier jar holds the module's native library and the FFmpeg shared libraries it uses. Applications that do not use the module therefore never download FFmpeg. Use the classifier of the platform you build for: `windows-x86_64`, `windows-aarch64`, `linux-x86_64`, `linux-aarch64`, `linux-aarch32`, `macos-x86_64` or `macos-aarch64`. ::: info -Unlike `webrtc-java`, which brings its natives along by itself, this module cannot: the natives are built by the module rather than by a separate one, so a dependency on them would have nothing to resolve against on a first build. Asking for them explicitly is the price of that. +Unlike `webrtc-java`, this module cannot bring its natives along by itself. The module builds its natives itself, not a separate module, so on a first build a dependency on them would have nothing to resolve against. That is why you ask for them explicitly. ::: ## Sending a File -`MediaFileSource` is the short way to do all of it. It opens the source, creates a media source for each kind of media the file actually has, and wires a player to feed them: +`MediaFileSource` does all of it in one step. It opens the source, creates a media source for each kind of media the file has, and connects a player that feeds them: ```java // Import required classes @@ -78,7 +71,7 @@ peerConnection.addTrack(audioTrack, List.of("stream")); source.play(); ``` -A file with no audio has no audio source, and likewise for video, so check before making a track: +A file without audio has no audio source, and a file without video has no video source. Check before you create a track: ```java if (source.getVideoSource() != null) { @@ -87,7 +80,11 @@ if (source.getVideoSource() != null) { } ``` -The audio of this module is pushed into its track, like that of a `CustomAudioSource`, so the same factory and peer connection can send the microphone beside it. A factory that sends only pushed audio never opens the recording device. +The module pushes its audio into the track, as a `CustomAudioSource` does. The same factory and peer connection can therefore send the microphone beside it. A factory that sends only pushed audio never opens the recording device. + +Decoding happens entirely in native code, and frames never pass through Java. The module hands decoded pictures straight to the native side of a `CustomVideoSource`, and audio in 10 ms chunks to a `CustomAudioSource`. + +Both are paced in real time and carry the presentation times of the file. A receiver therefore gets the timing of the media, not the timing of a Java thread. ## Reading What a Source Contains @@ -114,16 +111,20 @@ try (MediaReader reader = new MediaReader(Path.of("movie.mp4"))) { } ``` -The width and height are those of the video as it is shown, which can differ from what the file stores. A phone stores a portrait video sideways with a rotation next to it: its frames keep the stored size and carry the rotation in `VideoFrame.rotation`, which the video views and the receiving peers apply, and `MediaInfo` reports the upright size. A video of pixels that are not square, such as 720x576 PAL with 16:15 pixels, is scaled to square ones while it plays, to 768x576 in that case, because WebRTC has no way to describe them. One side grows so that no detail is lost, at the price of one copy of each picture. Only quarter turns are applied: a video that is to be shown mirrored, or turned by another angle, plays as it is stored. So does one whose pixels are said to be more than four times as wide as high, or the other way round. +The width and height are those of the video as it is shown. They can differ from what the file stores: -A `MediaFileSource` already has this, so there is no need to open the file twice: +- **Rotation:** A phone stores a portrait video sideways, with a rotation beside it. Its frames keep the stored size and carry the rotation in `VideoFrame.rotation`, which the video views and the receiving peers apply. `MediaInfo` reports the upright size. +- **Pixels that are not square,** such as those of 720x576 PAL with 16:15 pixels, are scaled to square ones while the video plays, here to 768x576. WebRTC has no way to describe them. One side grows, so that no detail is lost, at the price of one copy of each picture. +- **Exceptions:** Only quarter turns are applied. A video that is to be shown mirrored, or turned by another angle, plays as it is stored. So does a video whose pixels are said to be more than four times as wide as high, or the other way round. + +A `MediaFileSource` already has this information, so there is no need to open the file twice: ```java MediaInfo info = source.getInfo(); ``` ::: info -Duration is `0` for a source whose container does not say how long it runs, which is the case for live streams. Opening fails if a source holds nothing that can be played. +The duration is `0` for a source whose container does not say how long it runs, as is the case for live streams. Opening fails if a source holds nothing that can be played. ::: ## Controlling Playback @@ -138,13 +139,24 @@ source.getPositionUs(); // where playback has got to source.getState(); // IDLE, PLAYING, PAUSED, ENDED or CLOSED ``` -Looping keeps the timing across the seam, so a looping file works as a stand-in for a camera that never stops. A looping source never reports an end of stream. +Looping keeps the timing across the seam, so a looping file can stand in for a camera that never stops. A looping source never reports an end of stream. + +Positions count from the start of the source, which is its first video or audio frame, whichever comes first. The times a container gives its frames need not begin at zero. + +A file cut out of a longer recording, or an MPEG-TS capture, may start at any time, and `seek(2_000_000)` still means two seconds into it. Video and audio share that start, so a stream that begins after the other one keeps that distance. + +### Seeking + +`seek` starts playback at the position asked for. To produce a picture at all, the decoder has to go back to the keyframe at or before the position. The frames from there up to the position are decoded and dropped, and the audio is cut at the position. + +How long a seek takes therefore depends on how often the media was encoded with keyframes, not on how long the media is: -Positions count from the start of the source, which is its first video or audio frame, whichever comes first. The times a container gives its frames do not have to begin at zero: a file cut out of a longer recording or an MPEG-TS capture may start at any time, and `seek(2_000_000)` still means two seconds into it. Video and audio share that start, so a stream that begins after the other one keeps that distance. +- MKV and MP4 have an index that takes the reader straight to that keyframe. +- MPEG-TS has none. A search for the position by its timestamps can land past the keyframe before it. The player then searches again further back: one second before the position, then two, four and so on, so that the seek still starts at the position. -A seek starts playback at the position asked for. The decoder has to go back to the keyframe at or before it to produce a picture at all, so the frames from there up to the position are decoded and dropped, and the audio is cut at it. How long a seek takes therefore depends on how often the media was encoded with keyframes, not on how long the media is: MKV and MP4 have an index that takes the reader straight to that keyframe. MPEG-TS has none, and a search for the position by its timestamps can land past the keyframe before it. The player then searches again further back, a second before the position, then two, four and so on, so that the seek still starts at the position. +`fastSeek` moves to that keyframe and plays from there, without decoding up to the position. It takes no longer, however far apart the keyframes are. -`fastSeek` moves to that keyframe and plays from there, without decoding up to the position, so it takes no longer however far the keyframes are apart. That suits a slider being dragged, as browsers' `fastSeek()` does: fast seeks while it moves, and a `seek` once it is let go. In a container without an index, such as MPEG-TS, the keyframe can be the one after the position. +That suits a slider being dragged, as the `fastSeek()` of browsers does: fast seeks while it moves, and a `seek` once it is let go. In a container without an index, such as MPEG-TS, the keyframe can be the one after the position. ## Following Playback @@ -172,21 +184,30 @@ source.setListener(new MediaPlayerListener() { }); ``` -Output failures are reported through `onError`. A single rejected video frame or audio chunk is dropped without advancing the position for that item; playback continues. These reports are limited to one per second for video and one for audio. Ten consecutive failed deliveries of either kind stop playback in `PAUSED` and produce a final error report. An unexpected C++ exception on the decoder thread also stops playback in `PAUSED`. After either terminal failure, `play()` and `seek()` have no effect: close the player and create a new one to retry. +Output failures are reported through `onError`: -::: warning -Every call arrives on the player's own event thread, never on the thread that called `play()`, `pause()` or `close()`, and never on the one decoding the media. The calls come one at a time, in the order the player went through what they report, so a listener needs no locking of its own against itself. A command returns without waiting for its state change to be reported, so `getState()` can already read a later state than the call in progress reports. `close()` returns once the listener has been told of everything, `CLOSED` last. +- A single rejected video frame or audio chunk is dropped, without advancing the position for it. Playback continues. +- These reports are limited to one per second for video and one per second for audio. +- Ten failed deliveries in a row, of either kind, stop playback in `PAUSED` and produce a final error report. +- An unexpected C++ exception on the decoder thread also stops playback in `PAUSED`. -A listener must still return promptly: the calls behind it wait, and so does a `close()` from another thread. It must not wait on the player. +After either of these stops, `play()` and `seek()` have no effect. To retry, close the player and create a new one. -Closing the player, or the `MediaFileSource`, from a listener is fine: it takes effect at once, and the native player is released on another thread once the listener has returned. The same holds for a sink of a track made from the player's sources, which the player calls on threads of its own as well. +::: warning +- Every call arrives on the player's own event thread. It never arrives on the thread that called `play()`, `pause()` or `close()`, nor on the one decoding the media. +- The calls come one at a time, in the order the player went through what they report. A listener therefore needs no locking against itself. +- A command returns without waiting for its state change to be reported. `getState()` can therefore already read a later state than the call in progress reports. +- `close()` returns once the listener has been told of everything, `CLOSED` last. +- A listener must return promptly: the calls behind it wait, and so does a `close()` from another thread. It must not wait on the player. +- Closing the player, or the `MediaFileSource`, from a listener is fine. It takes effect at once, and the native player is released on another thread once the listener has returned. +- The same holds for a sink of a track made from the player's sources. The player calls sinks on threads of its own as well. ::: ## Memory -Decoding runs ahead of playback by up to sixty pictures, and at most by what `MediaPlayer.getMaxQueuedVideoBytes()` reports, which is 256 MiB. That keeps sixty pictures of 1080p, about twenty of 4K, and whatever the resolution, a player does not take more than that for pictures waiting to be delivered. Several players take that much each, on top of what the decoders use themselves. +Decoding runs ahead of playback by up to sixty pictures, and by no more than what `MediaPlayer.getMaxQueuedVideoBytes()` reports: 256 MiB. That holds sixty pictures of 1080p, or about twenty of 4K. Whatever the resolution, a player takes no more than that for pictures waiting to be delivered. -To see what an application really needs, ask the players that run: +Several players take that much each, on top of what the decoders use themselves. To see what an application really needs, ask the players that run: ```java source.getQueuedVideoBytes(); // what the queued pictures take right now @@ -195,7 +216,7 @@ source.getPeakQueuedVideoBytes(); // the most they have taken so far ## Feeding Your Own Media Sources -`MediaFileSource` creates the media sources for you. An application that needs its own — to feed a source it already created, or to send only the video of a file — uses `MediaPlayer` directly: +`MediaFileSource` creates the media sources for you. To feed a source you already created, or to send only the video of a file, use `MediaPlayer` directly: ```java // Import required classes @@ -214,14 +235,15 @@ player.play(); ``` ::: info -The player takes over the reader it is given. That reader must not be used or closed afterwards; closing the player releases it. - -It also keeps the native side of the media sources it feeds alive until it is closed, so disposing of a source or its track while the player runs is safe. A source that was already disposed is rejected with `IllegalStateException` before the reader is taken over; the caller can still use or close that reader. Source acquisition is protected against concurrent disposal: the constructor either acquires its own reference or rejects the disposed source. +- The player takes over the reader it is given. Don't use or close that reader afterwards; closing the player releases it. +- The player keeps the native side of the media sources it feeds alive until it is closed. Disposing of a source or its track while the player runs is therefore safe. +- A source that was already disposed is rejected with `IllegalStateException`, before the reader is taken over. The caller can then still use or close that reader. +- A source disposed while the constructor runs is safe too: the constructor either acquires its own reference to it or rejects it. ::: ## Decoding in Hardware -Decoding H.264, H.265/HEVC, VP9 and AV1 takes a good part of a processor at high resolutions, and for every stream of a camera wall. A player can decode them on the media engine or GPU of the machine instead. It does so when asked to, with a flag on `MediaFileSource` or `MediaPlayer`: +At high resolutions, and for every stream of a camera wall, decoding H.264, H.265/HEVC, VP9 and AV1 takes a good part of a processor. A player can decode them on the media engine or GPU of the machine instead. It does so when asked to, with a flag on `MediaFileSource` or `MediaPlayer`: ```java // A file whose video is decoded in hardware, where the platform can. @@ -235,14 +257,67 @@ MediaPlayer player = new MediaPlayer(new MediaReader(path), videoSource, null, t boolean hardware = source.getPlayer().isHardwareDecoding(); ``` -The pictures are the same ones software decoding gives; in the tests, every frame of media in each of these codecs is compared with its software counterpart (for HEVC, with checksums of the pictures FFmpeg's software decoder gave once). Hardware decoding is off unless asked for, except for H.265/HEVC. +Hardware decoding is off unless asked for, except for H.265/HEVC. The pictures are the same ones software decoding gives. The tests compare every frame of media in each of these codecs with its software counterpart; for HEVC, with checksums of the pictures that FFmpeg's software decoder gave once. + +::: warning HEVC is decoded in hardware only +A player decodes HEVC on the media engine or GPU whether or not the flag is set, and never in software. `isHardwareDecoding()` is always `true` for it. + +- If the machine has no hardware decoder for HEVC, creating the player fails with an `IOException`. +- If the hardware does not take a stream, such as a profile it cannot decode, or fails while playing, playback stops with an error. +- The [fallback to software](#fallback-to-software) does not apply to HEVC. +::: + +### Platforms + +| Platform | Decodes through | Needs | +|---|---|---| +| macOS | VideoToolbox | | +| Windows | Direct3D 12 | Windows 10 version 2004 or later, with a driver that decodes video through Direct3D 12 | +| Windows | Direct3D 11 | Used where Direct3D 12 is not available | +| Linux, NVIDIA GPUs | NVDEC | `libcuda.so.1` and `libnvcuvid.so.1`, which the driver provides | +| Linux, Intel and AMD GPUs | VA-API | `libva.so.2` and `libva-drm.so.2` with a driver: Intel's `iHD` from intel-media-driver, or Mesa's `radeonsi` | +| Linux, Intel and AMD GPUs | Vulkan Video, otherwise | The Vulkan loader `libvulkan.so.1` and a driver with the video extensions of the codec: Mesa's ANV or RADV | -**HEVC is decoded in hardware only.** A player decodes it on the media engine or GPU whether or not the flag is set, and never in software: `isHardwareDecoding()` is always `true` for it. Where the machine has no hardware decoder for HEVC, creating the player fails with an `IOException`; where the hardware does not take a stream (a profile it cannot decode) or fails while playing, playback stops with an error. The fallback below does not apply to it. +- Linux means x86-64 and ARM64. +- For Vulkan Video, Mesa's drivers must be recent enough for each codec, and AV1 and VP9 need the newest. Some distributions ship Mesa with its video extensions switched off. +- VA-API comes before Vulkan Video, because a Vulkan device can be created even where its driver does not decode, and the player picks a device by whether it can be created. +- These libraries are loaded only when a player asks for hardware, so a machine without them is not affected. +- The flag is accepted everywhere. Where the platform has no hardware decoder, or the FFmpeg build has none for it (32-bit ARM), the video is decoded in software, except HEVC. `isHardwareDecoding()` then says so. -- **Platforms:** macOS through VideoToolbox, Windows through Direct3D 12 (Windows 10 version 2004 or later, with a driver that decodes video through it) and otherwise Direct3D 11, and Linux (x86-64 and ARM64) through NVDEC on NVIDIA GPUs (the driver provides `libcuda.so.1` and `libnvcuvid.so.1`) and on the GPUs of Intel and AMD through VA-API (`libva.so.2` and `libva-drm.so.2` with a driver: Intel's `iHD` from intel-media-driver, or Mesa's `radeonsi`) and otherwise Vulkan Video (the Vulkan loader `libvulkan.so.1` and a driver with the video extensions of the codec: Mesa's ANV or RADV, recent enough for each codec, AV1 and VP9 needing the newest, which some distributions ship switched off). VA-API comes first because a Vulkan device can be created where its driver does not decode, and the player picks a device by whether it can be created. These libraries are loaded when a player asks for hardware, so a machine without them is not affected. The flag is accepted everywhere; where the platform has no hardware decoder, or the FFmpeg build has none for it (32-bit ARM), the video is decoded in software, except HEVC, and `isHardwareDecoding()` says so. -- **Codecs:** H.264, H.265/HEVC, VP9 and AV1, where the GPU decodes them: a GPU that is a few years old may lack AV1, and on a Mac only an M3 or later has it. VP8, MPEG-4 and MJPEG are decoded in software, and so is AV1 where there is no GPU decoder for it (it is decoded by dav1d, which is built into the FFmpeg libraries). -- **Fallback** (every codec but HEVC): a stream the hardware does not take, such as a profile it cannot decode, is decoded in software without the player noticing more than `isHardwareDecoding()` turning `false`. If the hardware fails before its first picture, the packets it was sent are decoded again in software and none is lost; if it fails after one, decoding goes on in software from the next key frame. A damaged packet is not taken for a hardware failure after the first picture: as with software decoding, playback stops with an error, and the hardware keeps the stream. -- **What it saves** is processor time, not the copy: a decoded picture is read back from the media engine into system memory, and converted to I420 as WebRTC wants it. On an Apple M2, a 1080p H.264 stream took 11.6 ms of processor time per frame in software and 1.4 ms in hardware, and a 4K stream 35.7 ms against 4.2 ms, about seven to eight times less. Per frame, software on all cores is faster on the clock, which does not matter at playback speed. Windows gains less: on an NVIDIA GeForce RTX 3080 through Direct3D 12, a 1080p H.264 stream took about 2.1 ms of processor time per frame against 3.6 ms in software, and a 4K stream 5.4 ms against 6.6 ms. Direct3D 11 reads the picture back more slowly and took more than software at 4K (12.4 ms), which is why Direct3D 12 comes first. These are figures for one machine each and for synthetic media, not a promise; try it on your own streams before relying on it. +### Codecs + +H.264, H.265/HEVC, VP9 and AV1 are decoded in hardware where the GPU decodes them. A GPU that is a few years old may lack AV1, and on a Mac only an M3 or later has it. + +VP8, MPEG-4 and MJPEG are decoded in software. So is AV1 where there is no GPU decoder for it. It is then decoded by dav1d, which is built into the FFmpeg libraries. + +### Fallback to Software + +For every codec but HEVC: + +- A stream the hardware does not take, such as a profile it cannot decode, is decoded in software. The only sign of it is `isHardwareDecoding()` turning `false`. +- If the hardware fails before its first picture, the packets it was sent are decoded again in software, and none is lost. +- If the hardware fails after its first picture, decoding goes on in software from the next key frame. +- After the first picture, a damaged packet is not taken for a hardware failure. As with software decoding, playback stops with an error, and the hardware keeps the stream. + +### What It Saves + +Hardware decoding saves processor time, not the copy. A decoded picture is read back from the media engine into system memory, and converted to I420 as WebRTC wants it. + +::: details Measurements +Processor time per frame: + +| Machine | Stream | Software | Hardware | +|---|---|---|---| +| Apple M2 | 1080p H.264 | 11.6 ms | 1.4 ms | +| Apple M2 | 4K H.264 | 35.7 ms | 4.2 ms | +| NVIDIA GeForce RTX 3080, Direct3D 12 | 1080p H.264 | 3.6 ms | about 2.1 ms | +| NVIDIA GeForce RTX 3080, Direct3D 12 | 4K H.264 | 6.6 ms | 5.4 ms | + +- On the M2, hardware takes about seven to eight times less. Windows gains less. +- Per frame, software on all cores is faster on the clock. At playback speed, that does not matter. +- Direct3D 11 reads the picture back more slowly and took more than software at 4K (12.4 ms). That is why Direct3D 12 comes first. +- These are figures for one machine each and for synthetic media, not a promise. Try it on your own streams before relying on it. +::: ## Playing a Live Stream @@ -252,9 +327,9 @@ A source can also be a live stream behind an `rtsp://` URL, which is what IP cam MediaFileSource source = new MediaFileSource("rtsp://camera.local:554/stream1"); ``` -Credentials go into the URL, as `rtsp://user:password@camera.local/stream1`. FFmpeg first asks the server for RTP over UDP, and falls back to RTP interleaved on the RTSP connection over TCP if the server refuses UDP or no UDP packets arrive, which is what gets a stream through most firewalls. +Credentials go into the URL, as in `rtsp://user:password@camera.local/stream1`. FFmpeg first asks the server for RTP over UDP. If the server refuses UDP, or no UDP packets arrive, it falls back to RTP interleaved on the RTSP connection over TCP, which gets a stream through most firewalls. -A network source can stall or disappear, so every operation that waits on a source has a time limit: opening it, and then each read during playback. It is 10 seconds unless you pass another: +A network source can stall or disappear, so every operation that waits on a source has a time limit: opening it, and then each read during playback. The limit is 10 seconds unless you pass another: ```java // Import required classes @@ -263,15 +338,22 @@ import java.time.Duration; MediaFileSource source = new MediaFileSource("rtsp://camera.local/stream1", Duration.ofSeconds(5)); ``` -`Duration.ZERO` waits as long as it takes, and is the only limit that does: a positive one waits at least a microsecond, and one too long for microseconds to count waits as long as they can count. A negative one is rejected. +- `Duration.ZERO` waits as long as it takes. It is the only limit that does. +- A positive limit waits at least a microsecond. One too long for microseconds to count waits as long as they can count. +- A negative limit is rejected. + +If the server cannot be reached, the constructor throws an `IOException` once the time is up. A stream that stops sending during playback is reported to the listener's `onError`, and the player is left `PAUSED`. Closing a player never waits for the time limit: it breaks off whatever the player is waiting for at once. -An unreachable server makes the constructor throw an `IOException` once that time is up. A stream that stops sending during playback is reported to the listener's `onError`, and the player is left `PAUSED`. Closing a player never waits for the time limit: it breaks off whatever the player is waiting for at once. +A live stream has no end and no length: -A live stream has no end and no length: its `MediaInfo` reports a duration of 0, looping does not apply, and seeking is reported as an error. Pausing stops reading from the server, but the server keeps sending; resuming carries on from where playback paused, which leaves it behind real time by as long as it was paused. To get back to live, close the source and open it again. +- Its `MediaInfo` reports a duration of 0. +- Looping does not apply, and seeking is reported as an error. +- Pausing stops reading from the server, but the server keeps sending. Resuming carries on from where playback paused, which leaves it behind real time by as long as it was paused. +- To get back to live, close the source and open it again. ## Closing -Closing a `MediaFileSource` stops playback and releases the player along with both media sources. Release the senders a peer connection handed out, and the tracks, before that: +Closing a `MediaFileSource` stops playback, and releases the player along with both media sources. Before that, release the senders a peer connection handed out, and the tracks: ```java videoSender.dispose(); @@ -291,23 +373,33 @@ The FFmpeg build is deliberately small, and carries only what this module plays: | **Video** | H.264, H.265/HEVC (hardware only), VP8, VP9, AV1, MPEG-4 (including Xvid and DivX), Microsoft MPEG-4 v1 to v3, MJPEG | | **Audio** | AAC, MP3, MP2, AC-3, Opus, Vorbis, FLAC, PCM, MS and IMA ADPCM | -Audio of any rate or layout is resampled to what WebRTC takes, which is 48 kHz 16-bit PCM in mono or stereo. Video that decodes to I420 — almost all 8-bit H.264, VP8, VP9 and MPEG-4 — reaches the encoder without being copied; anything else is converted first. +Audio of any rate or layout is resampled to what WebRTC takes: 48 kHz 16-bit PCM, in mono or stereo. Video that decodes to I420, which is almost all 8-bit H.264, VP8, VP9 and MPEG-4, reaches the encoder without being copied. Anything else is converted first. -Video retains its color matrix, full or limited range, color primaries and transfer function through conversion and into the native WebRTC frame, including hardware readback. Missing matrix/range metadata defaults to BT.601 limited range; RGB and JPEG pixel formats imply full range. RGB is converted to BT.601 YUV. The negotiated encoder and receiver determine how this information is carried and displayed. +Video keeps its color matrix, full or limited range, color primaries and transfer function through conversion and into the native WebRTC frame, also after hardware readback: -Sources can be local files, or live streams over RTSP (`rtsp://`), with RTP over UDP or TCP. Protocols that need TLS, such as `rtsps://` and `https://`, are not part of the build, and neither are HLS, DASH and RTMP. A source cannot reach any other protocol either, even one FFmpeg uses internally, so a URL passed on from a user cannot make the module fetch something else. +- Without matrix and range metadata, BT.601 limited range is assumed. +- RGB and JPEG pixel formats imply full range. +- RGB is converted to BT.601 YUV. + +The negotiated encoder and the receiver decide how this information is carried and displayed. + +Sources can be local files, or live streams over RTSP (`rtsp://`), with RTP over UDP or TCP. Protocols that need TLS, such as `rtsps://` and `https://`, are not part of the build, and neither are HLS, DASH and RTMP. + +A source cannot reach any other protocol either, not even one that FFmpeg uses internally. A URL passed on from a user therefore cannot make the module fetch something else. ## Licensing -The module uses FFmpeg under the LGPL version 2.1 or later. It is configured without `--enable-gpl` and without `--enable-nonfree`, and FFmpeg is linked dynamically and shipped as separate files inside the platform jar, so its libraries may be replaced with your own build, as the LGPL requires. The wrapper code is licensed under the Apache License 2.0 like the rest of webrtc-java. +The module uses FFmpeg under the LGPL version 2.1 or later. FFmpeg is configured without `--enable-gpl` and without `--enable-nonfree`. It is linked dynamically and shipped as separate files inside the platform jar, so its libraries may be replaced with your own build, as the LGPL requires. + +The wrapper code is licensed under the Apache License 2.0, like the rest of webrtc-java. -Each platform jar carries the LGPL text and a notice under `META-INF/licenses/ffmpeg`, naming the FFmpeg release the libraries are built from, unmodified, and where its source is. If you redistribute your application with these jars, keep those files with them. +Each platform jar carries the LGPL text and a notice under `META-INF/licenses/ffmpeg`. The notice names the FFmpeg release the libraries are built from, unmodified, and where its source is. If you redistribute your application with these jars, keep those files with them. Some of the formats FFmpeg decodes, such as H.264, H.265/HEVC and AAC, may be covered by patents in some countries. Whether your use of them needs a patent license is for you to determine. ## Complete Example -See `MediaFileExample` in the `webrtc-examples` module, which opens a file, reports what it contains, sends its tracks from one [Client API](/guide/client-api) peer to another and follows playback to the end. +See `MediaFileExample` in the `webrtc-examples` module. It opens a file, reports what it contains, sends its tracks from one [Client API](/guide/client-api) peer to another, and follows playback to the end. ```shell mvn -pl webrtc-examples compile From ab58e42aeec6fdaf2623b38dc6e9e7989c99541d Mon Sep 17 00:00:00 2001 From: Alex Andres Date: Thu, 12 Nov 2026 19:35:04 +0100 Subject: [PATCH 03/37] docs: write the end-to-end encryption page in shorter sentences Same content: long sentences split, rules and failure cases as lists. --- docs/guide/advanced/end-to-end-encryption.md | 127 ++++++++++++------- 1 file changed, 83 insertions(+), 44 deletions(-) diff --git a/docs/guide/advanced/end-to-end-encryption.md b/docs/guide/advanced/end-to-end-encryption.md index 4c7d2e09..f82f3985 100644 --- a/docs/guide/advanced/end-to-end-encryption.md +++ b/docs/guide/advanced/end-to-end-encryption.md @@ -1,18 +1,15 @@ # End-to-End Encryption -This guide explains how to encrypt a call end to end, so that the media servers, TURN servers and SFUs between the participants forward media they cannot read. It covers: +This guide explains how to encrypt a call end to end. The media servers, TURN servers and SFUs between the participants then forward media they cannot read. -- Encrypting a call with the client API, or with senders and receivers directly -- Participants, keys and key generations -- Rotating keys -- What happens to frames that cannot be encrypted or decrypted -- Replay protection -- The codecs that can be encrypted -- Recording an encrypted call -- Encrypting a call with a browser -- The frame format, as a specification for other implementations +The [encoded transforms](/guide/advanced/encoded-transforms) are where the encryption happens. On top of an encryption algorithm, `dev.onvoid.webrtc.e2ee` adds: -The [encoded transforms](/guide/advanced/encoded-transforms) are where the encryption happens. `dev.onvoid.webrtc.e2ee` adds what an encryption algorithm alone does not settle: the [SFrame](https://www.rfc-editor.org/rfc/rfc9605) format of RFC 9605, a key provider that holds the keys of every participant with their key IDs and generations, separate keys for every stream a participant sends, protection against replayed frames, a defined behavior for every frame that fails, and a recording policy. +- the [SFrame](https://www.rfc-editor.org/rfc/rfc9605) format of RFC 9605, +- a key provider that holds the keys of every participant, with their key IDs and generations, +- separate keys for every stream a participant sends, +- protection against replayed frames, +- a defined behavior for every frame that fails, +- a recording policy. ## Encrypting a Call @@ -58,31 +55,42 @@ The argument of `createEncryptor` and `createDecryptor` asks for a key frame whe ## Participants and Keys -A participant ID is a number from 0 to 232-1, which every participant of the call knows the others by, such as the index of a member in an [MLS](https://www.rfc-editor.org/rfc/rfc9420) group or a number the application's server hands out. Each participant encrypts with a key of its own, which the others decrypt with. This is the "sender keys" scheme of RFC 9605, Section 5.1, and it follows the rule of the RFC that a key is only ever used for encryption by one participant: `setDecryptionKey` refuses the local participant's ID. +A participant ID is a number from 0 to 232-1, by which every participant of the call knows the others. It can be the index of a member in an [MLS](https://www.rfc-editor.org/rfc/rfc9420) group, or a number the application's server hands out. -Every key belongs to a generation, 0 to 255, which goes up with each new key of a participant and wraps around after 255. A receiver keeps a participant's generations side by side, so a new key can be handed out before it is used. `generateKey()` makes a random key as long as the cipher suite's hash output; any key of at least 16 bytes works. +Each participant encrypts with a key of its own, and the others decrypt with it. This is the "sender keys" scheme of RFC 9605, Section 5.1. It follows the rule of the RFC that only one participant ever encrypts with a key: `setDecryptionKey` refuses the local participant's ID. + +Every key belongs to a generation from 0 to 255. The generation goes up with each new key of a participant, and wraps around after 255. A receiver keeps a participant's generations side by side, so a new key can be handed out before it is used. + +`generateKey()` makes a random key as long as the hash output of the cipher suite. Any key of at least 16 bytes works. ::: warning -Use a key for one call only, and never give it to a second `KeyProvider`. A key provider counts the frames it encrypts under each key, and a new one starts over, which would encrypt two frames under the same key and nonce. +Use a key for one call only, and never give it to a second `KeyProvider`. A key provider counts the frames it encrypts under each key, and a new key provider starts counting over. Two frames would then be encrypted under the same key and nonce. ::: ### Distributing Keys -The keys have to reach the other participants over a channel that the servers forwarding the media cannot read. That channel, and how participants authenticate each other on it, decides how secure the call is. Common choices are an MLS group, whose exporter gives a key per epoch (`MLS-Exporter("SFrame 1.0 Base Key", "", 16)`, RFC 9605, Section 5.2), or a messaging channel that is itself end-to-end encrypted. The signaling channel of the call is only good enough if the signaling server is trusted not to read it. +The keys have to reach the other participants over a channel that the servers forwarding the media cannot read. How secure the call is depends on that channel, and on how the participants authenticate each other on it. Common choices are: + +- an MLS group, whose exporter gives a key per epoch (`MLS-Exporter("SFrame 1.0 Base Key", "", 16)`, RFC 9605, Section 5.2), +- a messaging channel that is itself end-to-end encrypted. + +The signaling channel of the call is only good enough if the signaling server is trusted not to read it. ### Rotating Keys -A participant changes its key when someone joins or leaves the call, so that the newcomer cannot decrypt what was sent before, and the one who left cannot decrypt what is sent after. To rotate: +A participant changes its key when someone joins or leaves the call. The newcomer then cannot decrypt what was sent before, and the one who left cannot decrypt what is sent after. To rotate: 1. Make a new key with the next generation, and send it to the others, who `setDecryptionKey` it next to the old one. -2. Once they have it, acknowledged or after a timeout, `setEncryptionKey` it. Frames go out with the new key from the next one on. -3. Nothing more is needed on the receiving side. A receiver retires a participant's older generations 5 seconds after the first frame of a newer one decrypted (`Builder.keyRetention`), which covers the frames that were on their way during the switch. `removeDecryptionKey` and `removeParticipant` remove keys right away. +2. Once they have it, acknowledged or after a timeout, `setEncryptionKey` it. From the next frame on, frames go out with the new key. +3. Nothing more is needed on the receiving side. A receiver retires a participant's older generations 5 seconds after the first frame of a newer one decrypted (`Builder.keyRetention`). That covers the frames that were on their way during the switch. `removeDecryptionKey` and `removeParticipant` remove keys right away. -A video sender asks its encoder for a key frame with the first frame under a new key, so that a participant who has just joined, and has no key for the frames before, does not have to wait for the next one. +With the first frame under a new key, a video sender asks its encoder for a key frame. A participant who has just joined has no key for the frames before, so it does not have to wait for the next key frame. ### Streams -Every RTP stream a participant sends, its audio, its video and every simulcast layer, gets a context of its own, and with it a key ID and a derived key of its own. Streams count their frames independently and never share a key and nonce, without the application having to give them different keys. One encryptor may serve several senders, as the client API's peer options do, since it tells the streams apart by their SSRC. +Every RTP stream a participant sends gets a context of its own: its audio, its video and every simulcast layer. With the context comes a key ID and a derived key of its own. + +Streams count their frames independently and never share a key and nonce, without the application having to give them different keys. One encryptor may serve several senders, as it does for the client API's peer options, since it tells the streams apart by their SSRC. ## Failed Frames @@ -94,12 +102,14 @@ A frame that cannot be encrypted is dropped, never sent in the clear. A received | `UNKNOWN_KEY` | receiving | The key the frame was encrypted with is not there: not arrived yet, removed, or retired. | | `AUTHENTICATION` | receiving | The frame was changed on the way, or encrypted with another key of the same key ID. | | `REPLAY` | receiving | A frame with the same counter was decrypted before, or the frame is older than the replay window. | -| `MALFORMED` | both | A received frame is no SFrame ciphertext, as when the other side does not encrypt, or a frame to send is too short for its codec's header. | +| `MALFORMED` | both | Received: no SFrame ciphertext, as when the other side does not encrypt. Sent: too short for its codec's header. | | `UNSUPPORTED_CODEC` | both | H.264 or H.265, which the frame format does not cover (see [Codecs](#codecs)). | -A stream whose key has not arrived yet drops its frames until it has, and then decrypts again from the first frame whose key is there. Frames are not held back to wait for a key, since a frame is only valid while its transform runs. +A stream whose key has not arrived yet drops its frames until the key is there. It then decrypts again from the first frame whose key is there. Frames are not held back to wait for a key, since a frame is only valid while its transform runs. -The listener of the key provider hears of a stream when it begins to fail, and again only when it fails for another reason or after frames went through in between, so a stream without its key does not report each of its frames. The event says which participant and key generation the frames need, which is what a request for a missing key takes: +The listener of the key provider hears of a stream when the stream begins to fail. It hears of it again only when the stream fails for another reason, or when it fails again after frames went through in between. A stream without its key therefore does not report each of its frames. + +The event says which participant and key generation the frames need, which is what a request for a missing key takes: ```java KeyProvider keys = KeyProvider.builder(myId) @@ -115,29 +125,39 @@ The listener runs on the transform thread of the sender or receiver, so it shoul ### Key Frames -A decoder cannot decode the video frames that follow one it never got, so a video receiver that dropped frames asks the remote sender for a key frame as soon as frames decrypt again, at most twice a second. A video sender asks its own encoder for one when it starts to use a new key, and when it sends again after dropping frames for want of a key. +A decoder cannot decode the video frames that follow one it never got. A video receiver that dropped frames therefore asks the remote sender for a key frame as soon as frames decrypt again, at most twice a second. + +A video sender asks its own encoder for a key frame in two cases: when it starts to use a new key, and when it sends again after dropping frames for want of a key. ## Replay Protection -A receiver accepts each counter of a stream once: a frame that repeats the counter of one already decrypted, as a server that records and resends frames would produce, is dropped, and so is a frame older than the replay window. The window reaches 128 frames back by default, several seconds of any stream and far more than any reordering. `Builder.replayWindow(frames)` changes it, and 0 turns the protection off. A counter only counts as seen once its frame authenticated, so forged frames cannot move the window. +A receiver accepts each counter of a stream once. It drops a frame that repeats the counter of a frame already decrypted, as a server that records and resends frames would produce. It also drops a frame older than the replay window. + +The window reaches 128 frames back by default: several seconds of any stream, and far more than any reordering. `Builder.replayWindow(frames)` changes it, and 0 turns the protection off. A counter only counts as seen once its frame authenticated, so forged frames cannot move the window. ## Codecs -The frame format covers VP8, VP9, AV1 and every audio codec. WebRTC's packetizers and depacketizers read parts of some frames, so the format keeps those parts readable and authenticates them instead: +The frame format covers VP8, VP9, AV1 and every audio codec. WebRTC's packetizers and depacketizers read parts of some frames. The format therefore keeps those parts readable, and authenticates them instead of encrypting them: - **VP8** keeps its payload header in the clear: 10 bytes of a key frame, 3 of a delta frame. The receiver finds key frames and the frame size there. - **Opus** keeps its first byte, the TOC byte, in the clear, as browsers do. -- **AV1** is split into OBUs by its packetizer, so the encrypted frame travels as a single OBU: a sequence header for a key frame, which is how the receiver recognizes one, and a frame OBU otherwise. +- **AV1** is split into OBUs by its packetizer, so the encrypted frame travels as a single OBU. It is a sequence header for a key frame, which is how the receiver recognizes one, and a frame OBU otherwise. - **VP9** and other audio codecs are encrypted whole. -H.264 and H.265 frames are split at their start codes before they are sent, which a ciphertext would contain anywhere, so they are dropped with `UNSUPPORTED_CODEC`. A call has to negotiate them away, which matters on macOS in particular, where WebRTC prefers H.264 through VideoToolbox: +H.264 and H.265 frames are split at their start codes before they are sent. A ciphertext could contain a start code anywhere, so these frames are dropped with `UNSUPPORTED_CODEC`. A call has to negotiate them away. That matters on macOS in particular, where WebRTC prefers H.264 through VideoToolbox: -- A peer of the client API with `e2ee(keys)` does so by itself: it offers and answers only VP8, VP9 and AV1 video, in WebRTC's order. `videoCodecs(...)` of the [WHIP and WHEP](/guide/client/whip-whep) builders still picks among them, and fails the session with a `ConfigurationException` if it names none of them. +- A peer of the client API with `e2ee(keys)` does so by itself: it offers and answers only VP8, VP9 and AV1 video, in WebRTC's order. `videoCodecs(...)` of the [WHIP and WHEP](/guide/client/whip-whep) builders still picks among them. If it names none of them, the session fails with a `ConfigurationException`. - Without the client API, call `setCodecPreferences` on each video transceiver with the codecs for which `KeyProvider.supportsCodec(codec.getMimeType())` is true, keeping `rtx`, `red` and `ulpfec`. ## Recording -A [`MediaRecorder`](/guide/media/media-recording) records a sender's frames before its transform and a receiver's frames after it, so it would store an encrypted call in the clear. Whether that may happen is decided where the encryption is set up: the transforms of a key provider keep their frames from every observer of encoded frames, the recorder included, and `MediaRecorder.addTrack` refuses their senders, receivers and tracks with an `IllegalStateException`. A transform set on a track that is already being recorded keeps the recorder from any further frame of it. +A [`MediaRecorder`](/guide/media/media-recording) records a sender's frames before its transform, and a receiver's frames after it. It would therefore store an encrypted call in the clear. + +Whether that may happen is decided where the encryption is set up: + +- The transforms of a key provider keep their frames from every observer of encoded frames, the recorder included. +- `MediaRecorder.addTrack` refuses their senders, receivers and tracks with an `IllegalStateException`. +- A transform set on a track that is already being recorded keeps every further frame of that track from the recorder. An application that records its calls on purpose says so: @@ -147,11 +167,13 @@ KeyProvider keys = KeyProvider.builder(myId) .build(); ``` -A recording made this way is decrypted media on the disk of one participant, outside the protection of the call. Other transforms decide for themselves through `RTCEncodedFrameTransformer.allowsPlaintextObservers()`, which is true unless they say otherwise. +A recording made this way is decrypted media on the disk of one participant, outside the protection of the call. Other transforms decide for themselves, through `RTCEncodedFrameTransformer.allowsPlaintextObservers()`, which is true unless they say otherwise. ## Browsers -Browsers have no built-in SFrame support yet, but they run the same format with [`RTCRtpScriptTransform`](https://www.w3.org/TR/webrtc-encoded-transform/) and WebCrypto. [`webrtc-java-sframe.js`](/e2ee/webrtc-java-sframe.js) implements it, with a `KeyProvider` that works like the Java one. webrtc-java's tests run it in Node.js against the RFC's test vectors, and check that frames of every codec layout and cipher suite encrypted on either side decrypt on the other. +Browsers have no built-in SFrame support yet, but they run the same format with [`RTCRtpScriptTransform`](https://www.w3.org/TR/webrtc-encoded-transform/) and WebCrypto. [`webrtc-java-sframe.js`](/e2ee/webrtc-java-sframe.js) implements it, with a `KeyProvider` that works like the Java one. + +webrtc-java's tests run it in Node.js against the RFC's test vectors. They also check that frames of every codec layout and cipher suite, encrypted on either side, decrypt on the other. The transforms run in a worker: @@ -183,7 +205,7 @@ sender.transform = new RTCRtpScriptTransform(worker, { side: 'send' }); receiver.transform = new RTCRtpScriptTransform(worker, { side: 'receive' }); ``` -`attach` asks for key frames through the transformer as the Java side does. Browsers that predate `RTCRtpScriptTransform` need the older `createEncodedStreams()` instead, whose frames the encryptor's and decryptor's `transformFrame(frame, controller)` take just the same. +`attach` asks for key frames through the transformer, as the Java side does. Browsers that predate `RTCRtpScriptTransform` need the older `createEncodedStreams()` instead. Its frames go to the `transformFrame(frame, controller)` of the encryptor and decryptor just the same. ## Frame Format @@ -191,15 +213,21 @@ This section specifies the format, so that it can be implemented elsewhere. It i ### Version -This is version 1, named `webrtc-java-sframe/1`: `KeyProvider.FORMAT` in Java, `FORMAT` in the browser module. Nothing in a frame says which format it is in, so the participants of a call agree on it, as on the cipher suite and the keys: by the application's signaling, as part of what it sends with the keys. A later version, such as SFrame with the RTP payload format WebRTC is working on, which needs no codec layouts, has a name of its own and is agreed the same way, never told apart by trying. +This is version 1, named `webrtc-java-sframe/1`: `KeyProvider.FORMAT` in Java, `FORMAT` in the browser module. + +Nothing in a frame says which format it is in. The participants of a call therefore agree on it, as on the cipher suite and the keys: through the application's signaling, as part of what it sends with the keys. + +A later version has a name of its own and is agreed the same way, never told apart by trying. One example would be SFrame with the RTP payload format WebRTC is working on, which needs no codec layouts. ### Cipher Suites and Keys -Every participant of a call uses the same cipher suite of RFC 9605, Section 4.5, by default `AES_128_GCM_SHA256_128` (0x0004). The key and salt of each key ID are derived from the participant's key with `derive_key_salt` (RFC 9605, Section 4.4.2), and frames are encrypted and decrypted with `encrypt` and `decrypt` (Sections 4.4.3 and 4.4.4). +Every participant of a call uses the same cipher suite of RFC 9605, Section 4.5, by default `AES_128_GCM_SHA256_128` (0x0004). + +The key and salt of each key ID are derived from the participant's key with `derive_key_salt` (RFC 9605, Section 4.4.2). Frames are encrypted and decrypted with `encrypt` and `decrypt` (Sections 4.4.3 and 4.4.4). ### Key IDs -A key ID (KID) is made up of four fields, from the most significant bit down. The lowest 16 bits are the "key generation | ratchet step" of the sender keys scheme of RFC 9605, Section 5.1, and the fields above them follow the MLS scheme of Section 5.2: +A key ID (KID) is made up of four fields, from the most significant bit down. The lowest 16 bits are the "key generation | ratchet step" of the sender keys scheme of RFC 9605, Section 5.1. The fields above them follow the MLS scheme of Section 5.2: ``` 63 48 47 16 15 8 7 0 @@ -210,16 +238,18 @@ A key ID (KID) is made up of four fields, from the most significant bit down. Th - **participant**: the ID of the participant who encrypted the frame. A receiver finds the key by it and by the generation. - **generation**: the generation of that participant's key, 0 to 255. -- **ratchet step**: how many times the generation's key was moved forward with the ratchet of RFC 9605, Section 5.1, `base_key[i+1] = HKDF-Expand(HKDF-Extract("", base_key[i]), "SFrame 1.0 Ratchet", Nh)`. This version of the format always sends 0, and a receiver drops a frame with any other step as one whose key it does not have. -- **context**: one per RTP stream the participant sends, given out from 0 upwards and never reused for the participant's key provider, which therefore serves up to 65,536 streams. Since the KID goes into the key derivation, every stream encrypts with a key and salt of its own. +- **ratchet step**: how many times the generation's key was moved forward with the ratchet of RFC 9605, Section 5.1, `base_key[i+1] = HKDF-Expand(HKDF-Extract("", base_key[i]), "SFrame 1.0 Ratchet", Nh)`. This version of the format always sends 0. A receiver drops a frame with any other step, as one whose key it does not have. +- **context**: one per RTP stream the participant sends. Contexts are given out from 0 upwards and never reused within the participant's key provider, which therefore serves up to 65,536 streams. Since the KID goes into the key derivation, every stream encrypts with a key and salt of its own. ### Counters -Each stream counts its frames from 0, one counter value (CTR) per frame, and never resets the counter, not even when the key changes. A receiver accepts a counter at most once per KID, and only if it is at most the replay window below the highest counter it accepted. It marks a counter as seen only after the frame authenticated. +Each stream counts its frames from 0, one counter value (CTR) per frame. It never resets the counter, not even when the key changes. + +A receiver accepts a counter at most once per KID, and only if it is at most the replay window below the highest counter it accepted. It marks a counter as seen only after the frame authenticated. ### Frame Layout -A protected frame is the codec's clear prefix followed by an SFrame ciphertext, the header, the ciphertext and the tag: +A protected frame is the codec's clear prefix, followed by an SFrame ciphertext: the header, the ciphertext and the tag. ``` +--------------+---------------+------------------------+-----+ @@ -227,7 +257,7 @@ A protected frame is the codec's clear prefix followed by an SFrame ciphertext, +--------------+---------------+------------------------+-----+ ``` -The clear prefix is also the `metadata` that SFrame authenticates along with its header, and the plaintext is the rest of the frame: +The clear prefix is also the `metadata` that SFrame authenticates along with its header. The plaintext is the rest of the frame: | Codec (MIME type) | Clear prefix | | --- | --- | @@ -238,15 +268,24 @@ The clear prefix is also the `metadata` that SFrame authenticates along with its | any other `audio/*` | none | | anything else | not covered: the frame is dropped | -An AV1 frame arrives with its OBU header changed by the depacketizer, which sets the size bit (`0x02`) and inserts the OBU's size as a LEB128 value. A receiver accepts the OBU header with or without the size bit, skips the size, and authenticates the header without the size bit: `0x08` or `0x30`. It takes a VP8 frame for a key frame if its first byte has the lowest bit clear, as the depacketizer does. +An AV1 frame arrives with its OBU header changed by the depacketizer. The depacketizer sets the size bit (`0x02`) and inserts the OBU's size as a LEB128 value. A receiver accepts the OBU header with or without the size bit, and skips the size. It authenticates the header without the size bit: `0x08` or `0x30`. + +A receiver takes a VP8 frame for a key frame if its first byte has the lowest bit clear, as the depacketizer does. ### Receiving -A receiver drops a frame, rather than passing it to the decoder, when its codec is not covered, it is shorter than its clear prefix, its header is not in the minimal encoding of RFC 9605, Section 4.3, it has no key for the KID's participant and generation, its counter fails the replay check, or it does not authenticate. +A receiver drops a frame, rather than passing it to the decoder, if: + +- its codec is not covered, +- it is shorter than its clear prefix, +- its header is not in the minimal encoding of RFC 9605, Section 4.3, +- there is no key for the KID's participant and generation, +- its counter fails the replay check, or +- it does not authenticate. ## Security Considerations -- SFrame protects the media, not its metadata. The servers still see who sends, at what rate and with which key IDs, which tell the participant IDs and key generations, and they see the clear prefixes: whether a VP8 frame is a key frame and its size, and the Opus TOC byte. +- SFrame protects the media, not its metadata. The servers still see who sends, at what rate and with which key IDs, and the key IDs tell the participant IDs and key generations. The servers also see the clear prefixes: whether a VP8 frame is a key frame and its frame size, and the Opus TOC byte. - Every participant who has a key can encrypt frames that pass for that key's participant, since SFrame keys are symmetric (RFC 9605, Section 7.2). Telling participants apart is up to the key distribution. - Short tags (the `AES_128_CTR_HMAC_SHA256_*` suites) make forgeries more likely; keep the replay window on with them. - The keys are held in Java memory for as long as the key provider has them. From 3cfb8164699bb3d9973cf0e0bfc11ad942e75703 Mon Sep 17 00:00:00 2001 From: Alex Andres Date: Thu, 12 Nov 2026 19:36:23 +0100 Subject: [PATCH 04/37] docs: write the video codecs page in shorter sentences Long sentences split. Also corrects which factories offer H.265: the hardware factories on every platform where the GPU handles it, not only on Macs. --- docs/guide/advanced/video-codecs.md | 65 +++++++++++++++++++---------- 1 file changed, 44 insertions(+), 21 deletions(-) diff --git a/docs/guide/advanced/video-codecs.md b/docs/guide/advanced/video-codecs.md index 7e27a422..edb4f673 100644 --- a/docs/guide/advanced/video-codecs.md +++ b/docs/guide/advanced/video-codecs.md @@ -2,13 +2,17 @@ -A `PeerConnectionFactory` decides which video codecs it can send and receive through a video encoder factory and a video decoder factory. Unless told otherwise it uses the codecs built into the library. This guide explains how to replace or extend them: to add a codec WebRTC does not have, to use a hardware encoder reached through another library, or to limit the codecs a factory offers. +A `PeerConnectionFactory` decides which video codecs it can send and receive through a video encoder factory and a video decoder factory. Unless told otherwise, it uses the codecs built into the library. This guide explains how to replace or extend them: + +- to add a codec WebRTC does not have, +- to use a hardware encoder reached through another library, +- to limit the codecs a factory offers. The API lives in the `dev.onvoid.webrtc.media.video.codec` package. ## Built-in Codecs -`DefaultVideoEncoderFactory` and `DefaultVideoDecoderFactory` provide the codecs built into the library. On Windows and Linux these are VP8, VP9, AV1 and H.264 in software; on macOS they are those of WebRTC's default factories, with H.264 through VideoToolbox. Ask a factory for the exact list: +`DefaultVideoEncoderFactory` and `DefaultVideoDecoderFactory` provide the codecs built into the library. On Windows and Linux, these are VP8, VP9, AV1 and H.264 in software. On macOS, they are those of WebRTC's default factories, with H.264 through VideoToolbox. Ask a factory for the exact list: ```java for (VideoCodecInfo codec : new DefaultVideoEncoderFactory().getSupportedCodecs()) { @@ -18,7 +22,7 @@ for (VideoCodecInfo codec : new DefaultVideoEncoderFactory().getSupportedCodecs( ### Hardware Encoding -`HardwareVideoEncoderFactory` offers the codecs of `DefaultVideoEncoderFactory`, and H.265 where the hardware encodes it, and encodes on the GPU where the platform supports it: +`HardwareVideoEncoderFactory` offers the codecs of `DefaultVideoEncoderFactory`, plus H.265 where the hardware encodes it. It encodes on the GPU where the platform supports it: ```java PeerConnectionFactory factory = PeerConnectionFactory.builder() @@ -37,9 +41,9 @@ Which GPUs encode which codec, by platform and interface: | macOS | VideoToolbox | Apple | ✓ | Macs with a hardware HEVC encoder | – | – | | | **Software fallback** | | OpenH264 | none (3) | libvpx | libaom | -1. VP9 through Media Foundation has not been tested on such a GPU yet and is off by default; turn it on with a system property before the factory is created: `System.setProperty("webrtc.hardware.vp9Encoder", "true");` -2. Single layer, `L1T2` or `L1T3`, with a rate control of each layer where the driver has one that works (AMD's for H.265 and AV1, Intel's for VP9), otherwise one for all layers, which treats them alike. AMD's AV1 encoder takes 320x128 and up; smaller streams are encoded by libaom. -3. H.265 has no software encoder. It is offered after the other codecs, only where the hardware encodes it, and a call uses it only if it is preferred (e.g. with `RTCRtpTransceiver.setCodecPreferences()`) and the other end decodes it. On macOS, VideoToolbox never falls back to its software HEVC encoder. +1. VP9 through Media Foundation has not been tested on such a GPU yet, and is off by default. To turn it on, set a system property before the factory is created: `System.setProperty("webrtc.hardware.vp9Encoder", "true");` +2. Single layer, `L1T2` or `L1T3`. Each layer has a rate control of its own where the driver has one that works: AMD's for H.265 and AV1, Intel's for VP9. Otherwise one rate control covers all layers and treats them alike. AMD's AV1 encoder takes 320x128 and up; smaller streams are encoded by libaom. +3. H.265 has no software encoder. It is offered after the other codecs, and only where the hardware encodes it. A call uses it only if it is preferred, for example with `RTCRtpTransceiver.setCodecPreferences()`, and the other end decodes it. On macOS, VideoToolbox never falls back to its software HEVC encoder. The hardware encoders take over formats the software encoders offer too, so encoding in hardware never changes what is negotiated: @@ -50,7 +54,11 @@ The hardware encoders take over formats the software encoders offer too, so enco | VP9 | Profile 0 | | AV1 | Profile 0 | -Hardware encoders produce a single layer, but for the VA-API encoders, which encode temporal layers too (`L1T2` and `L1T3`) in every codec, so that an SFU can forward a lower frame rate. A stream that asks for spatial layers, or for temporal layers from another hardware encoder, is encoded in software. When a hardware encoder fails to start, for example because the GPU has no encoder sessions left, or fails while encoding, the stream switches to the next encoder in line (e.g. from NVENC to Media Foundation on Windows), and finally to the software encoder, and continues with a key frame. On a machine without a hardware encoder, the factory encodes like `DefaultVideoEncoderFactory`. +Hardware encoders produce a single layer. The exception are the VA-API encoders: they also encode temporal layers (`L1T2` and `L1T3`) in every codec, so that an SFU can forward a lower frame rate. A stream that asks for spatial layers, or for temporal layers from another hardware encoder, is encoded in software. + +A hardware encoder may fail to start, for example because the GPU has no encoder sessions left, or fail while encoding. The stream then switches to the next encoder in line, such as from NVENC to Media Foundation on Windows, and finally to the software encoder. It continues with a key frame. + +On a machine without a hardware encoder, the factory encodes like `DefaultVideoEncoderFactory`. Which encoder a stream uses shows in the `encoderImplementation` statistic of its `outbound-rtp` stats, e.g. `NVENC (NVIDIA GeForce RTX 4070)`, `MediaFoundation (AMDav1Encoder)`, `VA-API (Intel iHD driver ...)`, `VideoToolbox (H.265)`, `OpenH264` or `libaom`. @@ -75,7 +83,7 @@ Which GPUs decode which codec, by platform and interface; each where the GPU or | macOS | VideoToolbox | Apple | ✓ | Macs with a hardware HEVC decoder | Apple silicon | Apple M3 and newer | | | **Software fallback** | | FFmpeg (4) | none (3) | libvpx | dav1d | -1. The *HEVC Video Extensions* and *VP9 Video Extensions* come from the Microsoft Store (or, for H.265, an HEVC decoder of the GPU vendor); the *AV1 Video Extension* is part of Windows 11. Without the VP9 one, VP9 is decoded in software; without an HEVC decoder, H.265 is not offered. +1. The *HEVC Video Extensions* and *VP9 Video Extensions* come from the Microsoft Store. For H.265, an HEVC decoder of the GPU vendor works too. The *AV1 Video Extension* is part of Windows 11. Without the VP9 extension, VP9 is decoded in software; without an HEVC decoder, H.265 is not offered. 2. In 8 bit 4:2:0; see below for the streams that go to dav1d. 3. H.265 has no software decoder, so a stream the hardware cannot decode is not decoded. 4. On macOS, `DefaultVideoDecoderFactory` decodes H.264 through VideoToolbox already. @@ -93,7 +101,11 @@ Some streams are decoded in software even where the GPU decodes their codec: Temporal layers alone, as in `L1T3`, are a single spatial layer, and are decoded in hardware like any other stream. -A hardware decoder that fails, or turns out to decode in software, is replaced by the software decoder, which starts with the next key frame. Decoded frames are copied from GPU memory back to system memory, where WebRTC's frames are, so hardware decoding pays off mostly at high resolutions and with many streams; at low resolutions WebRTC's software decoders are about as cheap. On an Apple M2, for example, VP9 in hardware took a fraction of the processor time libvpx needs, while multi-threaded libvpx was about as fast on the clock. +A hardware decoder that fails, or turns out to decode in software, is replaced by the software decoder, which starts with the next key frame. + +Decoded frames are copied from GPU memory back to system memory, where WebRTC's frames are. Hardware decoding therefore pays off mostly at high resolutions and with many streams. At low resolutions, WebRTC's software decoders are about as cheap. + +On an Apple M2, for example, VP9 in hardware took a fraction of the processor time libvpx needs, while multi-threaded libvpx was about as fast on the clock. Which decoder a stream uses shows in the `decoderImplementation` statistic of its `inbound-rtp` stats, e.g. `MediaFoundation (Microsoft H264 Video Decoder MFT)`, `MediaFoundation (AV1VideoExtension)`, `VideoToolbox (VP9)`, `VideoToolbox (H.265)`, `VideoToolbox (AV1)`, `NVDEC (NVIDIA GeForce RTX 4070)` or `VA-API (Mesa Gallium driver ...)`. @@ -103,14 +115,18 @@ Which decoder a stream uses shows in the `decoderImplementation` statistic of it |---|---| | NVENC | NVIDIA driver 522 or newer on Windows, 520 or newer on Linux | | NVDEC | NVIDIA driver, which provides `libcuda.so.1` and `libnvcuvid.so.1` | -| VA-API | libva 2, a driver such as Intel's `intel-media-va-driver` (iHD) or Mesa's `mesa-va-drivers` for AMD that encodes H.264, and access to a render node in `/dev/dri` | +| VA-API | libva 2, a driver that encodes H.264 (see below), and access to a render node in `/dev/dri` | | Media Foundation | for decoding, the codec extensions above | -All of them are loaded at run time, so nothing needs to be installed on machines without them, and a machine without them encodes and decodes in software. +A VA-API driver is, for example, Intel's `intel-media-va-driver` (iHD), or Mesa's `mesa-va-drivers` for AMD. + +All of them are loaded at run time. Nothing needs to be installed on machines without them, and such a machine encodes and decodes in software. ### Listing Hardware Codecs -Both hardware factories tell which of their codecs the GPU of the machine handles. `getHardwareCodecs()` returns those of `getSupportedCodecs()`, in the same order, that a hardware encoder or decoder takes; on macOS that includes H.264 through VideoToolbox. The other supported codecs are encoded or decoded in software. The hardware is probed once per process, which every factory then shares, and again only when one of the system properties above is set differently; a GPU, driver or codec extension installed while the process runs is seen by the next one: +Both hardware factories tell which of their codecs the GPU of the machine handles. `getHardwareCodecs()` returns those of `getSupportedCodecs()` that a hardware encoder or decoder takes, in the same order. On macOS, that includes H.264 through VideoToolbox. The other supported codecs are encoded or decoded in software. + +The hardware is probed once per process, and every factory shares the result. It is probed again only when one of the system properties above is set differently. A GPU, driver or codec extension installed while the process runs is seen by the next process: ```java HardwareVideoEncoderFactory encoders = new HardwareVideoEncoderFactory(); @@ -122,11 +138,13 @@ for (VideoCodecInfo codec : encoders.getSupportedCodecs()) { } ``` -The list is per format: on Windows, for example, Media Foundation encoders take H.264 Constrained Baseline and Baseline with packetization mode 1, so the Main profile and packetization mode 0 are listed as software. A hardware codec is the one tried first, not a guarantee: a stream that falls back to software, as described above, or asks for what the hardware does not do, such as SVC, is encoded or decoded in software. The `powerEfficientEncoder` and `powerEfficientDecoder` statistics of the `outbound-rtp` and `inbound-rtp` stats tell what a running stream uses. +The list is per format. On Windows, for example, Media Foundation encoders take H.264 Constrained Baseline and Baseline with packetization mode 1, so the Main profile and packetization mode 0 are listed as software. + +A hardware codec is the one tried first, not a guarantee. A stream that falls back to software, as described above, or asks for what the hardware does not do, such as SVC, is encoded or decoded in software. The `powerEfficientEncoder` and `powerEfficientDecoder` statistics of the `outbound-rtp` and `inbound-rtp` stats tell what a running stream uses. ### Native Codecs -The encoders and decoders these factories create are `NativeVideoEncoder`s and `NativeVideoDecoder`s. They are placeholders that make WebRTC create the built-in codec, which then runs entirely inside WebRTC, so their methods are not to be called from Java. +The encoders and decoders these factories create are `NativeVideoEncoder`s and `NativeVideoDecoder`s. They are placeholders that make WebRTC create the built-in codec. The codec then runs entirely inside WebRTC, so their methods are not to be called from Java. ## Setting the Factories @@ -175,7 +193,12 @@ The list is in order of preference. A codec name WebRTC does not know, such as ` ### H.265 -The library is built with WebRTC's H.265 support: H.265 is negotiated with its SDP parameters (`profile-id`, `tier-flag`, `level-id`, `tx-mode`) and sent with the H.265 RTP payload format (RFC 7798). WebRTC has no H.265 encoder or decoder of its own, and this library adds none in software, so H.265 is offered only by a factory that brings one: the hardware factories on Macs that encode or decode it in hardware (see above), or one of your own, such as a hardware encoder reached through another library: +The library is built with WebRTC's H.265 support. H.265 is negotiated with its SDP parameters (`profile-id`, `tier-flag`, `level-id`, `tx-mode`) and sent with the H.265 RTP payload format (RFC 7798). + +WebRTC has no H.265 encoder or decoder of its own, and this library adds none in software. H.265 is therefore offered only by a factory that brings one: + +- the hardware factories, where the GPU encodes or decodes H.265 (see the tables above), +- a factory of your own, such as one with a hardware encoder reached through another library: ```java Map parameters = new HashMap<>(); @@ -187,7 +210,7 @@ parameters.put("tx-mode", "SRST"); codecs.add(new VideoCodecInfo("H265", parameters)); ``` -The encoder hands WebRTC each picture as an access unit in Annex B form, with the parameter sets (VPS, SPS and PPS) in every key frame, and the decoder receives them the same way. +The encoder hands WebRTC each picture as an access unit in Annex B form, with the parameter sets (VPS, SPS and PPS) in every key frame. The decoder receives them the same way. ## Implementing an Encoder @@ -239,10 +262,10 @@ class MyEncoder implements VideoEncoder { Things to keep in mind: -- The capture time of an `EncodedImage` must be the `timestampNs` of the frame it encodes. That is how WebRTC matches output to input; an image with any other capture time is dropped. -- The frame passed to `encode()` is valid only until the method returns. An encoder that works asynchronously calls `frame.retain()` and later `frame.release()`, and hands its output to the callback from its own thread, in encoding order. The pixels must not be changed, since other consumers may share them. +- The capture time of an `EncodedImage` must be the `timestampNs` of the frame it encodes. That is how WebRTC matches output to input. An image with any other capture time is dropped. +- The frame passed to `encode()` is valid only until the method returns. An encoder that works asynchronously calls `frame.retain()`, later `frame.release()`, and hands its output to the callback from its own thread, in encoding order. The pixels must not be changed, since other consumers may share them. - `onEncodedFrame()` copies the payload, so the buffer may be reused once it returns. Frames handed over after `release()` are ignored. -- `getScalingSettings()`, `getEncoderInfo()`, `getResolutionBitrateLimits()`, `getImplementationName()` and `isHardwareEncoder()` have defaults and may be overridden. The default scaling settings use WebRTC's quantizer thresholds for VP8, VP9, H.264 and H.265 (which has those of H.264 until WebRTC has its own); a custom codec without a quantizer should return `ScalingSettings.OFF`. +- `getScalingSettings()`, `getEncoderInfo()`, `getResolutionBitrateLimits()`, `getImplementationName()` and `isHardwareEncoder()` have defaults and may be overridden. The default scaling settings use WebRTC's quantizer thresholds for VP8, VP9, H.264 and H.265. H.265 uses those of H.264 until WebRTC has its own. A custom codec without a quantizer should return `ScalingSettings.OFF`. ## Implementing a Decoder @@ -279,11 +302,11 @@ class MyDecoder implements VideoDecoder { - The image and its buffer are read-only and valid only until `decode()` returns; copy the data to keep it longer. - A decoded frame must carry the capture time of the image it was decoded from as its timestamp. -- `onDecodedFrame()` takes a reference of its own, so the caller still releases the frame it created. A buffer whose `toI420()` returns a `NativeI420Buffer` is shared; any other is copied, and has to provide direct byte buffers as large as its strides and height require. +- `onDecodedFrame()` takes a reference of its own, so the caller still releases the frame it created. A buffer whose `toI420()` returns a `NativeI420Buffer` is shared. Any other buffer is copied, and has to provide direct byte buffers as large as its strides and height require. ## Errors -Whatever an encoder, decoder or factory method throws goes to the calling thread's uncaught exception handler, is logged, and is treated as `VideoCodecStatus.ERROR`, and a factory that throws from `createEncoder()` or `createDecoder()` creates no codec for that stream. On an error other than `FALLBACK_SOFTWARE` and `UNINITIALIZED`, WebRTC releases the codec and initializes it again. +Whatever an encoder, decoder or factory method throws goes to the calling thread's uncaught exception handler, and is logged. It is treated as `VideoCodecStatus.ERROR`. A factory that throws from `createEncoder()` or `createDecoder()` creates no codec for that stream. On an error other than `FALLBACK_SOFTWARE` and `UNINITIALIZED`, WebRTC releases the codec and initializes it again. ## Related API From 0c001df46a28afc0531dd0c7d7aa04597d7d9978 Mon Sep 17 00:00:00 2001 From: Alex Andres Date: Thu, 12 Nov 2026 19:37:45 +0100 Subject: [PATCH 05/37] docs: write the media recording page in shorter sentences Same content, with one point made exact: a video track without a key frame within three seconds starts the new segment at its next key frame. --- docs/guide/media/media-recording.md | 97 +++++++++++++++++------------ 1 file changed, 56 insertions(+), 41 deletions(-) diff --git a/docs/guide/media/media-recording.md b/docs/guide/media/media-recording.md index 2e934f6a..0c6ab15a 100644 --- a/docs/guide/media/media-recording.md +++ b/docs/guide/media/media-recording.md @@ -1,20 +1,10 @@ # Media Recording -This guide explains how to record what a peer connection sends or receives into a media file with `MediaRecorder`, from the `webrtc-java-media` module. It covers: +`MediaRecorder`, from the `webrtc-java-media` module, records what a peer connection sends or receives into a media file. See [Media Files](/guide/media/media-files) for how to add the module to a project. -- Recording senders and receivers -- Replacing a file that exists -- Choosing the container -- Splitting a recording into segments, and how many to keep -- Keeping the last seconds in memory and saving them on demand -- Following a recording with a listener -- How tracks are started, timed and left out -- Measuring what a recording lost -- Recording an end-to-end encrypted call +The recorder writes the encoded frames of each sender or receiver into the file as they are, without decoding or re-encoding them. A recording therefore costs next to no CPU, keeps exactly the quality that went over the network, and needs no codec, only the container. -`MediaRecorder` writes the encoded frames of each sender or receiver into the file as they are, without decoding or re-encoding them. A recording therefore costs next to no CPU, keeps exactly the quality that went over the network, and needs no codec at all, only the container. The frames are taken in native code, straight from WebRTC, and written by a thread of the recorder's own, so neither Java nor a slow disk ever holds up the call. - -See [Media Files](/guide/media/media-files) for how to add the module to a project. +The frames are taken in native code, straight from WebRTC, and written by a thread of the recorder's own. Neither Java nor a slow disk ever holds up the call. ## Recording a Call @@ -32,16 +22,16 @@ try (MediaRecorder recorder = new MediaRecorder(Paths.get("call.mkv"))) { } ``` -A sender's frames are recorded as they leave the encoder, a receiver's as they go to the decoder. Senders and receivers of any number of peer connections can go into one file, e.g. both sides of a call. +A sender's frames are recorded as they leave the encoder, a receiver's as they go to the decoder. Senders and receivers of any number of peer connections can go into one file, for example both sides of a call. -`stop()` waits until everything received so far is written and the file is finished and closed, and returns whether the recording was kept. A recorder that never got any media to write deletes what it wrote rather than leave an unplayable file behind. `close()` stops a recording that still runs. +`stop()` waits until everything received so far is written and the file is finished and closed. It returns whether the recording was kept. A recorder that never got any media to write deletes what it wrote, rather than leave an unplayable file behind. `close()` stops a recording that still runs. -How the recording ended, `getResult()` tells once the recorder is stopped, and keeps telling after `close()`: +Once the recorder is stopped, `getResult()` tells how the recording ended, also after `close()`: | Result | File | Meaning | | --- | --- | --- | | `COMPLETE` | kept | Everything was written, and the file was finished and closed. | -| `PARTIAL` | kept | Media went into the file, but writing, finishing or closing it failed later, or it could not [replace an existing file](#replacing-a-file). What made it into the file is kept, since it may still play. | +| `PARTIAL` | kept | Media went into the file, but then writing, finishing, closing or [replacing the existing file](#replacing-a-file) failed. What is in the file may still play. | | `EMPTY` | deleted | No media arrived before the stop. | | `FAILED` | deleted | Writing failed before any media went into the file. | @@ -64,11 +54,11 @@ recorder.addTrack(videoSender); // a Sender from peer.addTrack recorder.addTrack(remoteTrack); // a RemoteTrack from peer.onTrack ``` -The peer keeps its native senders and receivers to itself and asks them for key frames on its own thread, so nothing needs to be kept undisposed: removing a sender or closing the peer ends the track in the recording, and `stop()` still finishes the file. +The peer keeps its native senders and receivers to itself, and asks them for key frames on its own thread. Nothing needs to be kept undisposed: removing a sender or closing the peer ends the track in the recording, and `stop()` still finishes the file. ## Replacing a File -A recorder does not touch a file that exists: its constructor throws a `FileAlreadyExistsException`, so a new recording cannot cost an earlier one by accident. To replace the file, ask for it: +A recorder does not touch a file that exists: its constructor throws a `FileAlreadyExistsException`. A new recording therefore cannot cost an earlier one by accident. To replace the file, ask for it: ```java try (MediaRecorder recorder = new MediaRecorder(Paths.get("call.mkv"), true)) { @@ -76,9 +66,17 @@ try (MediaRecorder recorder = new MediaRecorder(Paths.get("call.mkv"), true)) { } ``` -The recording then goes into a temporary file beside it, named `.recording-.mkv`, and the existing file stays untouched until `stop()` has finished the recording. Only then does the recording take its place, in one step, with the permissions of the file it replaces, and a symbolic link stays a link to the new recording. A recording that gets no media, or fails before any went into the file, leaves the existing file as it was. +The recording then goes into a temporary file beside it, named `.recording-.mkv`. The existing file stays untouched until `stop()` has finished the recording. Only then does the recording take its place: + +- in one step, +- with the permissions of the file it replaces, +- and a symbolic link stays a link, now to the new recording. + +A recording that gets no media, or fails before any went into the file, leaves the existing file as it was. -If the process ends before the stop, the existing file is still there, next to the temporary file with what was recorded so far. If the recording cannot take the place of the existing file when it stops, it stays in the temporary file: `onError` is told, the result is `PARTIAL`, and `getFile()` returns where it is. +If the process ends before the stop, the existing file is still there. The temporary file beside it holds what was recorded so far. + +If the recording cannot take the place of the existing file when it stops, it stays in the temporary file. `onError` is told, the result is `PARTIAL`, and `getFile()` returns where the recording is. ## Containers @@ -90,13 +88,13 @@ The container follows the file name: | `.webm` | VP8, VP9, AV1 | Opus | Plays in browsers. | | `.mp4` | VP9, AV1, H.264, H.265 | Opus | Written fragmented, so it plays up to where a recording was cut short. | -A name FFmpeg does not know gets Matroska. A track whose codec the file cannot hold, VP8 in MP4 for example, is left out and reported as a warning, and the rest is recorded. +A name FFmpeg does not know gets Matroska. A track whose codec the file cannot hold, such as VP8 in MP4, is left out and reported as a warning. The rest is recorded. Matroska has no codec ID of its own for G.711, so it is stored as Windows audio (`A_MS/ACM`), the way FFmpeg itself stores it. Players built on FFmpeg, such as VLC and mpv, play it; some others do not. ## Recording in Segments -For recordings that run for hours, such as surveillance or support sessions, a recorder can split the recording into segments: files that each play on their own, ended by duration or size, of which it keeps as many as asked. +A recording that runs for hours, such as surveillance or a support session, can be split into segments. Each segment is a file that plays on its own, and ends at a duration or size. The recorder keeps as many segments as asked. ```java SegmentOptions segments = SegmentOptions.builder(Paths.get("recordings", "camera-%05d.mkv")) @@ -121,17 +119,21 @@ try (MediaRecorder recorder = new MediaRecorder(segments)) { } ``` -The file name of the pattern holds the segment's number as `%d`, or zero-padded as `%05d`; a literal percent sign is `%%`. The extension picks the container of all segments, as for a single file. Numbering goes on from the highest number already in the directory, so a recorder started again continues after the last segment, and a segment never empties a file that exists: its number is skipped instead. +The file name of the pattern holds the segment's number as `%d`, or zero-padded as `%05d`; a literal percent sign is `%%`. The extension picks the container of all segments, as for a single file. + +Numbering goes on from the highest number already in the directory, so a recorder started again continues after the last segment. A segment never empties a file that exists: its number is skipped instead. -- **Every segment begins at a key frame.** Once a segment has reached its duration or size, the recorder asks the video tracks for key frames, and the next segment begins with the first that arrives, usually a fraction of a second later. A segment is therefore somewhat longer or larger than the limit. Without video, a segment ends right at the limit. -- **Several video tracks lose nothing.** A video track whose key frame comes after the one that began the next segment goes on in the segment before until then, so neither file has a gap. One that sends no key frame within three seconds starts the new segment at its next. +- **Every segment begins at a key frame.** Once a segment has reached its duration or size, the recorder asks the video tracks for key frames. The next segment begins with the first that arrives, usually a fraction of a second later. A segment is therefore somewhat longer or larger than the limit. Without video, a segment ends right at the limit. +- **Several video tracks lose nothing.** Every other video track goes on in the segment before until its own key frame arrives, so neither file has a gap. A track that sends no key frame within three seconds starts in the new segment at its next key frame, and nothing of it is recorded until then. - **Each segment starts at zero.** Its tracks are shifted by the same amount, so they stay in sync, and one segment follows the other without a gap. `MediaRecorderSegment.getStart()` tells where a segment lies in the recording. -- **A codec change starts a segment.** A file cannot change the codec of a stream, so a track that changes its codec, e.g. after a renegotiation, goes on in the next segment, which begins as soon as the new codec is known. A single file leaves such a track out from then on instead. -- **A slow disk holds nothing up.** Segments are finished and opened by the recorder's writer thread, with the frames queuing up meanwhile as always, and old segments are deleted on the listener's thread. +- **A codec change starts a segment.** A file cannot change the codec of a stream. A track that changes its codec, for example after a renegotiation, therefore goes on in the next segment, which begins as soon as the new codec is known. A single file leaves such a track out from then on instead. +- **A slow disk holds nothing up.** The recorder's writer thread finishes and opens segments, while the frames queue up as always. Old segments are deleted on the listener's thread. + +`onSegmentFinished` hears of every segment once it is finished and closed, the last one when the recorder stops. Right after it returns, the oldest segments beyond `maxSegments` or `maxTotalSize` are deleted. A segment to be kept for longer has to be moved or copied by then. -`onSegmentFinished` hears of every segment once it is finished and closed, the last one when the recorder stops. Right after it returns, the oldest segments beyond `maxSegments` or `maxTotalSize` are deleted, so a segment to be kept for longer has to be moved or copied by then. The newest segment is always kept, and the one being written does not count toward `maxTotalSize`, so the recording takes up to that much and one segment more. `getSegments()` lists the segments that are kept, and `getFile()` returns the last one finished. +The newest segment is always kept. The one being written does not count toward `maxTotalSize`, so the recording takes up to that much and one segment more. `getSegments()` lists the segments that are kept, and `getFile()` returns the last one finished. -A write that fails, e.g. because the disk is full, ends the recording as it does for a single file. The segment it happened in is kept as `PARTIAL` if it holds media; one that failed before any went into it is deleted. +A write that fails, for example because the disk is full, ends the recording as it does for a single file. The segment it happened in is kept as `PARTIAL` if it holds media. A segment that failed before any media went into it is deleted. ## Replay Buffer @@ -150,9 +152,13 @@ try (ReplayBuffer buffer = new ReplayBuffer(Duration.ofSeconds(30))) { } ``` -`save` creates the file at once, refusing one that exists, and the buffer's own thread writes the replay, while the buffer goes on taking frames. The future completes on the listener's thread with a `MediaRecorderSegment`, or exceptionally if nothing could be saved, and the file is then deleted. A buffer can save any number of replays, in any container; a track whose codec the container cannot hold is left out of that replay with a warning. +`save` creates the file at once, and refuses one that exists. The buffer's own thread writes the replay, while the buffer goes on taking frames. -A replay begins with a key frame of every video track: the last one before the window, so a replay holds at least the window, and a few seconds more. Since WebRTC rarely sends key frames of its own accord, the buffer asks its video tracks for one every third of the window, at least every two seconds and at most every ten. Besides the window, the buffer holds at most 256 MB, or what its constructor is given; beyond that, the oldest frames go first, and a replay is shorter. A track that changes its codec is kept with the new codec from then on. +The future completes on the listener's thread with a `MediaRecorderSegment`. If nothing could be saved, it completes exceptionally, and the file is deleted. A buffer can save any number of replays, in any container. A track whose codec the container cannot hold is left out of that replay, with a warning. + +A replay begins with a key frame of every video track: the last one before the window. A replay therefore holds at least the window, and a few seconds more. WebRTC rarely sends key frames of its own accord, so the buffer asks its video tracks for one every third of the window, at least every two seconds and at most every ten. + +Besides the window, the buffer holds at most 256 MB, or what its constructor is given. Beyond that, the oldest frames go first, and a replay is shorter. A track that changes its codec is kept with the new codec from then on. ## Following a Recording @@ -178,23 +184,30 @@ recorder.setListener(new MediaRecorderListener() { }); ``` -Calls arrive in order on a thread the recorder keeps for them, so a listener may take its time and may even stop the recorder. +Calls arrive in order, on a thread the recorder keeps for them. A listener may therefore take its time, and may even stop the recorder. + +`onError` also hears of a file that cannot be finished or closed when the recorder stops, for example because the last data does not fit on the disk. The recording then ends as `PARTIAL`. -`onError` also hears of a file that cannot be finished or closed when the recorder stops, e.g. because the last data does not fit on the disk. The recording then ends as `PARTIAL`. +Running out of native memory while writing ends the recording the same way: -Running out of native memory while writing ends the recording the same way: `onError` is told, the frames still waiting are dropped, and the file is finished right away, so what is in it still plays. The recording ends as `PARTIAL`, or as `FAILED` if nothing had gone into the file yet. +- `onError` is told. +- The frames still waiting are dropped. +- The file is finished right away, so what is in it still plays. +- The recording ends as `PARTIAL`, or as `FAILED` if nothing had gone into the file yet. ## How Tracks Are Recorded -- **Video starts at a key frame.** A decoder can do nothing with the frames before one, so they are skipped. The recorder asks the sender or receiver for a key frame, so recording starts within moments instead of waiting until WebRTC sends one of its own accord, which it rarely does. -- **The file begins once the tracks are known.** The file header describes every stream, and a stream is only known from its first frames. The file begins once every track has sent some, or once the tracks that have waited three seconds for the rest. A track that sends nothing by then is left out, with a warning. +- **Video starts at a key frame.** A decoder can do nothing with the frames before one, so they are skipped. The recorder asks the sender or receiver for a key frame. Recording therefore starts within moments, instead of waiting until WebRTC sends one of its own accord, which it rarely does. +- **The file begins once the tracks are known.** The file header describes every stream, and a stream is only known from its first frames. The file begins once every track has sent some, or once the tracks have waited three seconds for the rest. A track that sends nothing by then is left out, with a warning. - **Tracks share one timeline.** Each track is placed by when its first frame arrived, and follows its own RTP timestamps from there, so its timing is exact. The tracks of one sender line up within the jitter of the network. - **One simulcast layer.** A sender with simulcast is recorded at the layer whose frames reach the recorder first. - **Nothing is lost to a slow disk, up to a point.** Frames queue up in memory for the writer, up to 64 MB. Beyond that, frames are dropped, and video starts over at the next key frame. `getStats()` [tells how much that cost](#measuring-what-was-lost). ## Measuring What Was Lost -A result of `COMPLETE` means that everything the recorder took went into the file. It says nothing about frames it had to drop before that, and a video gap can come from a single one, since video only goes on from the next key frame. `getStats()` tells what happened, per track and in the order the tracks were added. It can be asked at any time, also after `close()`: +A result of `COMPLETE` means that everything the recorder took went into the file. It says nothing about frames the recorder had to drop before that. A single dropped frame can leave a video gap, since video only goes on from the next key frame. + +`getStats()` tells what happened, per track, in the order the tracks were added. It can be asked at any time, also after `close()`: ```java MediaRecorderStats stats = recorder.getStats(); @@ -215,13 +228,15 @@ System.out.println("Queue peak: " + stats.getMaxQueuedBytes() + " bytes"); | `getDroppedAwaitingKeyFrame()` | Video frames dropped on the way to the next key frame, since nothing could decode them after what was lost before. | | `getDroppedFrames()` | All of these together. | -`getMaxQueuedBytes()` is the most that ever waited for the writer, of all tracks together, and shows how close a recording came to the limit of 64 MB. Frames a track was never meant to record do not count, such as those before its first key frame or those of another simulcast layer. +`getMaxQueuedBytes()` is the most that ever waited for the writer, of all tracks together. It shows how close a recording came to the limit of 64 MB. + +Frames a track was never meant to record do not count, such as those before its first key frame, or those of another simulcast layer. ## Recording an Encrypted Call -A sender's frames are recorded before an [encoded transform](/guide/advanced/encoded-transforms) runs on them, and a receiver's after it. An end-to-end encrypted call would therefore be recorded in the clear, on either side, so whether it may be is up to the encryption: +A sender's frames are recorded before an [encoded transform](/guide/advanced/encoded-transforms) runs on them, and a receiver's after it. An end-to-end encrypted call would therefore be recorded in the clear, on either side. Whether it may be is up to the encryption: -- The transforms of a `KeyProvider` of [End-to-End Encryption](/guide/advanced/end-to-end-encryption) do not allow it by default. `addTrack` refuses their senders, receivers and tracks with an `IllegalStateException`, and a recording that is running gets no further frame of a track once such a transform is set on it. +- The transforms of a `KeyProvider` of [End-to-End Encryption](/guide/advanced/end-to-end-encryption) do not allow it by default. `addTrack` refuses their senders, receivers and tracks with an `IllegalStateException`. A recording that is running gets no further frame of a track once such a transform is set on it. - A key provider built with `allowPlaintextRecording(true)` allows it, for an application that records its calls on purpose: ```java From 471bc70ef933e10ac052a8b8eab69453a72c3637 Mon Sep 17 00:00:00 2001 From: Alex Andres Date: Fri, 13 Nov 2026 01:38:51 +0100 Subject: [PATCH 06/37] docs: write the statistics page in shorter sentences Same content: long table cells and rules moved into lists below the tables. --- docs/guide/client/statistics.md | 127 +++++++++++++++----------------- 1 file changed, 60 insertions(+), 67 deletions(-) diff --git a/docs/guide/client/statistics.md b/docs/guide/client/statistics.md index ac362b12..46dda55b 100644 --- a/docs/guide/client/statistics.md +++ b/docs/guide/client/statistics.md @@ -18,15 +18,19 @@ peer.onStats(Duration.ofSeconds(1), stats -> System.out.printf( | `roundTripTime()` | The current round trip time on `transport()`, or null while not measured. | | `bytesSent()`, `bytesReceived()` | The bytes sent and received on all transports so far. | | `timestamp()` | When WebRTC measured the statistics. | -| `outgoingBitrate()`, `incomingBitrate()` | Bits per second between the measurement times of the previous sample and this one, or -1 when there is none to compare with: for a peer's first sample, after the clock went back, or after the byte counter started again. A report WebRTC hands out again within its short cache lifetime repeats the previous bitrates. | +| `outgoingBitrate()`, `incomingBitrate()` | Bits per second between the measurement times of the previous sample and this one, or -1 (see below). | | `packetLoss()` | The share of received packets lost so far, 0 to 1. After a long session it barely moves. | -| `recentPacketLoss()` | The share of packets lost of those expected since the previous sample, 0 to 1, and 0 if none were expected; -1 for the first sample and after the clock went back. A stream that came up or whose counters started again counts all it has. | +| `recentPacketLoss()` | The share of expected packets lost since the previous sample, 0 to 1; 0 if none were expected, or -1 (see below). | | `outboundVideo()`, `inboundVideo()` | Codec, resolution and frame rate of each video stream, and what limits a sent one. | -| `inboundAudio()`, `outboundAudio()` | Audio streams ordered by statistics id: codec, jitter, concealment counters, audio level and sample counters. Unavailable fields are null; received PCM callbacks are observed only while an `onAudio` sink is attached. | +| `inboundAudio()`, `outboundAudio()` | Audio streams ordered by statistics id: codec, jitter, concealment counters, audio level and sample counters. | | `diagnostics()` | The current state and confirmed `QualityCondition` causes. | | `diagnosticReport()` | A JSON snapshot with allowlisted diagnostics, media measurements, codec names, OS/architecture, Java version, library version and processor count. | | `report()` | The whole `RTCStatsReport`, including potentially sensitive connection information. | +- The bitrates are -1 when there is nothing to compare with: for a peer's first sample, after the clock went back, or after the byte counter started again. A report that WebRTC hands out again within its short cache lifetime repeats the previous bitrates. +- `recentPacketLoss()` is -1 for the first sample and after the clock went back. A stream that came up, or whose counters started again, counts all it has. +- In the audio statistics, unavailable fields are null. Received PCM callbacks are observed only while an `onAudio` sink is attached. + How `transport()` is chosen when media do not share one transport is described under [Details](#separate-transports). ## Diagnosis and Quality Events @@ -50,38 +54,35 @@ peer.stats().thenAccept(stats -> supportExecutor.execute(() -> { })); ``` -The export example uses `java.nio.file.Files`, `java.nio.file.Paths`, -`java.nio.charset.StandardCharsets` and `java.io.IOException`; `supportExecutor` -is an application-owned executor. +The export example uses `java.nio.file.Files`, `java.nio.file.Paths`, `java.nio.charset.StandardCharsets` and `java.io.IOException`. `supportExecutor` is an executor of the application. + +`onDiagnostics` reports the first diagnosis, and after that only changes of its state or causes, including recovery. Its handlers run on the peer's event executor, and closing the subscription stops sampling. All `stats`, `onStats` and `onDiagnostics` calls of a peer share one history. A captured `PeerStats` and its diagnosis never change. -`onDiagnostics` emits the initial diagnosis and then only changes to its state or -causes, including recovery. Handlers run on the peer's event executor; closing the -subscription stops sampling. All `stats`, `onStats` and `onDiagnostics` calls share -one history per peer. A captured `PeerStats` and its diagnosis stay immutable. +There is no combined score. A diagnosis is in one of three states: -There is no combined score. `OBSERVING` means there is no continuous measurement -interval yet (including after a clock rollback or a gap longer than five seconds). -`HEALTHY` means no supported problem has been confirmed; it is not a guarantee -that all media paths work. `DEGRADED` includes the following confirmed causes: +- `OBSERVING`: there is no continuous measurement interval yet. This includes the time after a clock rollback or a gap longer than five seconds. +- `HEALTHY`: no supported problem has been confirmed. It is not a guarantee that all media paths work. +- `DEGRADED`: one or more of these causes are confirmed: | Condition | Evidence | |---|---| | `VIDEO_CPU_LIMITED` | The same outgoing video RTP stream consistently reports `qualityLimitationReason=cpu`. | -| `HIGH_PACKET_LOSS` | Recent inbound packet loss is at least 5%; recovery needs at most 2%. Values between those thresholds retain the current state and restart the confirmation window. | -| `AUDIO_PACKETS_WITHOUT_PCM` | The same received audio RTP stream gains packets while no PCM callbacks reach its attached `onAudio` sinks. This observes sink ingress, before application queues and handlers; it does not diagnose speakers or slow application handlers. | +| `HIGH_PACKET_LOSS` | Recent inbound packet loss is at least 5%; recovery needs at most 2%. Values in between keep the state and restart the confirmation window. | +| `AUDIO_PACKETS_WITHOUT_PCM` | The same received audio RTP stream gains packets while no PCM callbacks reach its attached `onAudio` sinks. | -`TURN_IN_USE` is immediate information when either selected candidate on any -transport is a relay. It does not make a peer degraded. `TransportStats.relayed()` -also exposes this fact, while `candidateType()` continues to describe the local -candidate. +`AUDIO_PACKETS_WITHOUT_PCM` observes what reaches the sinks, before the application's queues and handlers. It does not diagnose speakers or slow application handlers. + +`TURN_IN_USE` is reported at once, as information, when either selected candidate on any transport is a relay. It does not make a peer degraded. `TransportStats.relayed()` also tells this, while `candidateType()` still describes the local candidate. How a condition is confirmed, the audio statistics and what the JSON export holds are described under [Details](#confirming-a-condition). ## Checking Before a Call -Diagnostics watch a call that is under way. `checkCall` answers the question before it: am I ready -for the call? It checks that the camera delivers frames, that the microphone delivers audio that is -neither silent nor clipped, and that a connection to a test peer comes up, all at the same time: +Diagnostics watch a call that is under way. `checkCall` answers the question before it: am I ready for the call? It checks, all at the same time, that: + +- the camera delivers frames, +- the microphone delivers audio that is neither silent nor clipped, +- a connection to a test peer comes up. ```java LocalCamera camera = rtc.media().camera(); @@ -105,34 +106,32 @@ rtc.checkCall(check).thenAccept(result -> { }); ``` -Every part is optional, but a check needs at least one. Each waits up to the check's `timeout` -(10 seconds by default) for the first frame, the first audio or the connection, then measures for -its `duration` (3 seconds by default). The future completes on an event thread of the runtime's -local media, so the rule against `join()` holds there. The tracks stay the application's: the check -neither closes nor mutes them, and the same tracks can go into the call once it is done. +Every part is optional, but a check needs at least one. Each part waits up to the check's `timeout` (10 seconds by default) for the first frame, the first audio or the connection. It then measures for the check's `duration` (3 seconds by default). + +The future completes on an event thread of the runtime's local media, so the [rule against `join()`](/guide/client/threads#don-t-wait-in-a-handler) holds there. The tracks stay the application's: the check neither closes nor mutes them, and the same tracks can go into the call once it is done. | Problem | Found when | |---|---| | `NO_VIDEO` | The video track delivered no frame within the timeout. | -| `DARK_VIDEO` | No frame had an average luma above 8% of white: a covered lens, a closed privacy shutter or no light. The brightest frame counts, since a camera's first frames are often dark while its exposure settles. A muted track sends black frames, so it is dark too. | +| `DARK_VIDEO` | No frame had an average luma above 8% of white: a covered lens, a closed privacy shutter or no light. | | `NO_AUDIO` | The audio track delivered no audio within the timeout. | | `SILENT_AUDIO` | The audio never rose above -50 dBFS: a muted microphone, or nobody spoke. | | `CLIPPED_AUDIO` | More than 0.1% of the samples were at full scale (99% of it or above): the input is too loud and distorts. | | `NO_CONNECTION` | No connection to the test peer came up within the timeout. `network().failure()` tells why: a `TimeoutException`, or the failure of the peer. | -Besides the problems, the result has the measurements behind them: frames, frame rate, resolution and -brightness of the video; how much audio arrived, with its peak and RMS level and the share of -clipped samples; and for the network the time it took to connect, the round trip time, the local -candidate type and whether the connection is relayed. `isReady()` means that no part found a -problem, not that the call is sure to work. +For `DARK_VIDEO`, the brightest frame counts, since a camera's first frames are often dark while its exposure settles. A muted track sends black frames, so it is dark too. + +Besides the problems, the result holds the measurements behind them: + +- **Video:** frames, frame rate, resolution and brightness. +- **Audio:** how much audio arrived, its peak and RMS level, and the share of clipped samples. +- **Network:** the time it took to connect, the round trip time, the local candidate type, and whether the connection is relayed. -Whether the network lets a call through cannot be told from the local configuration, only from a -connection that is made. The network part therefore connects to a real test peer: one the -application runs for this, such as an echo bot behind its own signaling, or any other peer that -answers the offer. The check creates a peer with the given options, opens a data channel so that it -offers, reads the connection's statistics once it is up and closes the peer. With `relayOnly(true)` -the connection can only go through the TURN servers of the options, so a connection proves that they -work; without it, a direct connection may come up instead. +`isReady()` means that no part found a problem, not that the call is sure to work. + +Whether the network lets a call through cannot be told from the local configuration, only from a connection that is made. The network part therefore connects to a real test peer. That is one the application runs for this, such as an echo bot behind its own signaling, or any other peer that answers the offer. + +The check creates a peer with the given options, and opens a data channel so that the peer offers. Once the connection is up, the check reads its statistics and closes the peer. With `relayOnly(true)`, the connection can only go through the TURN servers of the options, so a connection proves that they work. Without it, a direct connection may come up instead. How the microphone is checked without being heard is described under [Details](#how-the-microphone-is-checked). @@ -140,41 +139,35 @@ How the microphone is checked without being heard is described under [Details](# ### Separate Transports -A peer bundles its media on one transport by default. When the remote side refuses to bundle, each m-line has a transport of its own. `transport()` is then the one with a candidate pair in use that carried the most bytes, sent and received, and of transports that carried as many, the one with the lowest id. Read `transports()` to watch each connection by itself. +A peer bundles its media on one transport by default. If the remote side refuses to bundle, each m-line has a transport of its own. `transport()` is then the one with a candidate pair in use that carried the most bytes, sent and received. Of transports that carried as many, it is the one with the lowest id. Read `transports()` to watch each connection by itself. ### Confirming a Condition -Each problem and its recovery need at least three distinct measurements spanning -at least three seconds. Polling once a second is recommended; cached reports do -not advance the window. A missing stream or required measurement, a changed sink -set, or a reset audio counter discards that stream's evidence instead of guessing. -Sampling gaps longer than five seconds and backwards timestamps discard temporal -evidence. Diagnostics do not add audio sinks: without an application subscription, -PCM availability is unknown. Silence (audio level zero) alone is not a fault. +Each problem, and its recovery, needs at least three distinct measurements spanning at least three seconds. Polling once a second is recommended; cached reports do not advance the window. + +- A missing stream or required measurement, a changed set of sinks, or a reset audio counter discards that stream's evidence, instead of guessing. +- Gaps between samples longer than five seconds, and timestamps that go backwards, discard the evidence gathered over time. +- Diagnostics do not add audio sinks. Without a subscription of the application, whether PCM arrives is unknown. +- Silence (audio level zero) alone is not a fault. ### Audio Statistics -`AudioStats.jitter()` is a `Duration`, `audioLevel()` is linear (0 to 1), and -`concealedSamples()`, `concealmentEvents()` and `totalSamplesReceived()` are -cumulative counters. Outbound level is read from the media source when available; -receive-only measurements on outbound audio remain null. `pcmCallbacks()` counts -sink callbacks across subscriptions, so multiple subscriptions can count the same -audio more than once. +- `AudioStats.jitter()` is a `Duration`. +- `audioLevel()` is linear, from 0 to 1. +- `concealedSamples()`, `concealmentEvents()` and `totalSamplesReceived()` are cumulative counters. +- The outbound level is read from the media source when available. Measurements that exist only for received audio remain null on outbound audio. +- `pcmCallbacks()` counts sink callbacks across subscriptions, so several subscriptions can count the same audio more than once. ### The JSON Export -The JSON export includes the confirmation policy and cause explanations. It uses -schema version 1 and an explicit allowlist, including a -bounded set of codec MIME names; unknown names become `other`. It excludes raw -statistics, track/stream/transport/candidate identifiers, addresses, ports, URLs, -ICE credentials, SDP, certificates and codec parameters. Missing measurements -become JSON null. The raw `report()` and its string representation are not -sanitized diagnostic exports. +The JSON export includes the confirmation policy and explanations of the causes. It uses schema version 1 and an explicit allowlist, which includes a bounded set of codec MIME names; unknown names become `other`. Missing measurements become JSON null. + +It leaves out raw statistics, identifiers of tracks, streams, transports and candidates, addresses, ports, URLs, ICE credentials, SDP, certificates and codec parameters. + +The raw `report()` and its string representation are not sanitized like the export. ### How the Microphone Is Checked -Audio is measured as the track sends it, after audio processing. WebRTC records a microphone only -while it sends it, so for a microphone the check sends the track to a peer of its own, in the same -runtime, which drops every encoded frame before it goes out: nothing is played back, and the user -does not hear themselves. That peer, and the one of the network check, show among `rtc.peers()` -while the check runs and are closed before the result goes out. +Audio is measured as the track sends it, after audio processing. WebRTC records a microphone only while it sends it. For a microphone, the check therefore sends the track to a peer of its own, in the same runtime, which drops every encoded frame before it goes out. Nothing is played back, and the user does not hear themselves. + +That peer, and the one of the network check, show among `rtc.peers()` while the check runs. They are closed before the result goes out. From f1386d34b4c81c4c06d59cbae00e5a6a68162568 Mon Sep 17 00:00:00 2001 From: Alex Andres Date: Fri, 13 Nov 2026 01:39:45 +0100 Subject: [PATCH 07/37] docs: write the sending media page in shorter sentences Also corrects the audio of a media file: it is pushed audio, which works in every audio mode, not only CUSTOM and HEADLESS. --- docs/guide/client/sending-media.md | 39 ++++++++++++++++++++++-------- 1 file changed, 29 insertions(+), 10 deletions(-) diff --git a/docs/guide/client/sending-media.md b/docs/guide/client/sending-media.md index 5c322bfb..431338f3 100644 --- a/docs/guide/client/sending-media.md +++ b/docs/guide/client/sending-media.md @@ -23,7 +23,7 @@ Sender audioSender = peer.addTrack(microphone); | `screen(ScreenOptions)` | A screen or a window, as the [Desktop Video](/guide/video/desktop-capture) guide describes. | | `customVideo(String)` | Frames you push. | | `customAudio(String)` | Audio you push, in every mode. | -| `videoTrack(String, VideoTrackSource)`, `audioTrack(String, AudioTrackSource)` | A source of the core API, such as one the [media module](/guide/media/media-files) feeds, or a [`SystemAudioSource`](/guide/audio/system-audio-source) for the audio the system plays. The source stays yours. | +| `videoTrack(String, VideoTrackSource)`, `audioTrack(String, AudioTrackSource)` | A source of the core API, such as one the [media module](/guide/media/media-files) feeds, or a [`SystemAudioSource`](/guide/audio/system-audio-source). The source stays yours. | ```java Devices devices = media.devices(); @@ -65,7 +65,11 @@ peer.addTrack(rtc.media().audioTrack("movie", file.getAudioSource())); file.play(); ``` -Create the runtime before the file: the runtime loads the native library that the file's sources need. The sources belong to the `MediaFileSource`, not to the tracks, so close the runtime, or the tracks, before the file: `track.closeAsync()` returns a future that completes once the track let go of its source. The file's audio source is pushed audio, so the runtime needs the `CUSTOM` or `HEADLESS` audio mode. +Create the runtime before the file: the runtime loads the native library that the file's sources need. + +The sources belong to the `MediaFileSource`, not to the tracks. Close the runtime, or the tracks, before the file. `track.closeAsync()` returns a future that completes once the track has let go of its source. + +The file's audio source is pushed audio, so it works in every audio mode, also beside the microphone of a `DEVICES` runtime. ## Pushing Video and Audio @@ -86,9 +90,9 @@ CustomAudio voice = media.customAudio("voice"); voice.push(pcm, PcmFormat.of(48000, 1)); ``` -`CustomAudio` takes PCM in pieces of any length and from any thread. It cuts the audio into the 10 ms chunks WebRTC works with and sends one every 10 ms from a thread of its own, so pushing a whole second at once is as fine as pushing every 10 ms. `queuedMillis()` tells how much is still waiting, and `clear()` drops it. +`CustomAudio` takes PCM in pieces of any length and from any thread. It cuts the audio into the 10 ms chunks WebRTC works with, and sends one every 10 ms from a thread of its own. Pushing a whole second at once is therefore as fine as pushing every 10 ms. `queuedMillis()` tells how much is still waiting, and `clear()` drops it. -A producer faster than real time would otherwise fill memory and push the delay up without end, so the queue holds at most 10 seconds, and a `push` beyond that fails with an `AudioQueueFullException`, keeping none of its audio. `queueLimit(max, overflow)` sets another limit and what happens beyond it: +A producer faster than real time would otherwise fill memory, and push the delay up without end. The queue therefore holds at most 10 seconds. A `push` beyond that fails with an `AudioQueueFullException`, and none of its audio is kept. `queueLimit(max, overflow)` sets another limit, and what happens beyond it: ```java // Live audio: stay at most 200 ms behind, dropping the oldest audio. @@ -140,7 +144,9 @@ camera.switchCamera(media.devices().cameras().get(1)) .thenRun(() -> System.out.println("Now sending " + camera.device().getName())); ``` -The new camera keeps the resolution and frame rate of the old one, as far as it supports them. The switch opens the new camera before it closes the old one, so that the peers see no gap. Some devices cannot capture with two cameras at once, such as the two sensors of a laptop camera; then the old camera closes first, and the switch takes about half a second. What happens when a switch fails is described under [Details](#when-a-camera-switch-fails). +The new camera keeps the resolution and frame rate of the old one, as far as it supports them. The switch opens the new camera before it closes the old one, so that the peers see no gap. + +Some devices cannot capture with two cameras at once, such as the two sensors of a laptop camera. Then the old camera closes first, and the switch takes about half a second. What happens when a switch fails is described under [Details](#when-a-camera-switch-fails). ## Devices @@ -156,22 +162,35 @@ devices.useSpeaker(devices.speakers().get(0)); `useMicrophone` and `useSpeaker` switch devices also for tracks that are sent or played now. A device that was disconnected since the list was read fails the returned future, and the device in use stays as it was. On Linux, where WebRTC's audio device module cannot tell devices apart, the default device is used whichever is given. -A track does not end by itself when its device is disconnected or a captured window closes. `LocalTrack.onEnded` follows only from closing the track, or from a camera switch that can open neither camera. `onChange` tells of a device that went away; closing the track, or switching a camera to another device, is up to you. Nothing tells of a captured window that closes. +A track does not end by itself when its device is disconnected or a captured window closes: + +- `LocalTrack.onEnded` follows only from closing the track, or from a camera switch that can open neither camera. +- `onChange` tells of a device that went away. Closing the track, or switching a camera to another device, is up to you. +- Nothing tells of a captured window that closes. ## Details ### Timing of Pushed Audio -That thread does nothing else, so a slow push into WebRTC delays only the audio, not the timeouts of the runtime. Should it stall, as in a long garbage collection pause, it makes up for at most the last 50 ms instead of sending the whole pause at once; the queued audio then stays behind by the rest, until `clear()` drops it. +The thread that sends pushed audio does nothing else. A slow push into WebRTC therefore delays only the audio, not the timeouts of the runtime. + +If the thread stalls, as in a long garbage collection pause, it makes up for at most the last 50 ms, instead of sending the whole pause at once. The queued audio then stays behind by the rest, until `clear()` drops it. ### DTMF Tones -Tones still queued from an earlier call are replaced, and that call's future is cancelled. A video sender, or one whose remote peer accepts no telephone events, fails the future with an `IllegalStateException`. So do tones WebRTC stops sending midway, for example because the sender was paused with `configure`, a few seconds after their planned end. +- Tones still queued from an earlier call are replaced, and that call's future is cancelled. +- A video sender, or a sender whose remote peer accepts no telephone events, fails the future with an `IllegalStateException`. +- So do tones that WebRTC stops sending midway, for example because the sender was paused with `configure`. The future then fails a few seconds after their planned end. ### When a Camera Switch Fails -If the new camera cannot be opened, the future fails with a `MediaException`, and the track keeps sending the camera it had. If a peer does not take the new camera within the operation timeout, the future fails with a `WebRtcException`; the track goes back to the old camera, or keeps the new one when the old one had to close first. A track that closed its camera first and can open neither camera ends. +- If the new camera cannot be opened, the future fails with a `MediaException`, and the track keeps sending the camera it had. +- If a peer does not take the new camera within the operation timeout, the future fails with a `WebRtcException`. The track goes back to the old camera, or keeps the new one when the old one had to close first. +- A track that closed its camera first, and can open neither camera, ends. ### Device Lists -Where the system reports no device changes, as on a Linux machine without PulseAudio, they are read on each call instead. `screens()` and `windows()` ask the system on each call. A query that fails, including an unavailable audio backend, throws a `MediaException`, so an empty list means there are no such devices. Its cause is an `AudioBackendUnavailableException` when the audio backend is missing or unreachable, or a `DeviceException` for other native device failures. +- Where the system reports no device changes, as on a Linux machine without PulseAudio, the lists are read on each call instead. +- `screens()` and `windows()` ask the system on each call. +- A query that fails, also for an unavailable audio backend, throws a `MediaException`. An empty list therefore means there are no such devices. +- The cause of that `MediaException` is an `AudioBackendUnavailableException` when the audio backend is missing or unreachable, and a `DeviceException` for other native device failures. From c99a9655a3ecc24bdbf73d9a8a1da54867103927 Mon Sep 17 00:00:00 2001 From: Alex Andres Date: Fri, 13 Nov 2026 01:40:54 +0100 Subject: [PATCH 08/37] docs: write the peers and signaling page in shorter sentences Same content: recovery rules and JSON rules as lists. --- docs/guide/client/peers.md | 93 ++++++++++++++++++++++---------------- 1 file changed, 53 insertions(+), 40 deletions(-) diff --git a/docs/guide/client/peers.md b/docs/guide/client/peers.md index c22b12e8..3b1b40b3 100644 --- a/docs/guide/client/peers.md +++ b/docs/guide/client/peers.md @@ -39,7 +39,11 @@ Signaling signaling = message -> socket.send(message.toJson()); socket.onText(text -> peer.signal(SignalMessage.fromJson(text))); ``` -A transport class can implement both directions by overriding `Signaling.attach(Consumer)`, which the peer calls once when it is created, with the consumer to push received messages into. It returns a `Subscription` that removes that consumer, and only that one: the peer closes it when it closes, after its last `send`, on its event thread. The transport itself stays open, so a WebSocket can outlive the peer and take the next one, or serve several peers at once. A transport that keeps the consumer must drop it there, or the closed peer and the runtime with it stay reachable: +A transport class can implement both directions by overriding `Signaling.attach(Consumer)`. The peer calls it once, when the peer is created, with the consumer to push received messages into. + +`attach` returns a `Subscription` that removes that consumer, and only that one. The peer closes the subscription when it closes, after its last `send`, on its event thread. The transport itself stays open, so a WebSocket can outlive the peer and take the next one, or serve several peers at once. + +A transport that keeps the consumer must drop it when the subscription is closed. Otherwise the closed peer, and the runtime with it, stay reachable: ```java @Override @@ -103,7 +107,9 @@ Peer callee = rtc.createPeer(PeerOptions.builder().signaling(link.second()).poli ## Polite and Impolite Peers -With automatic negotiation, a peer sends an offer whenever its tracks or channels change. When both peers send one at the same moment, the **polite** peer drops its own offer and answers the other one, and the impolite peer ignores the offer it got. Exactly one of two peers must be polite. Peers are impolite by default, so set `polite(true)` on one side, for example on the peer that joined second. +With automatic negotiation, a peer sends an offer whenever its tracks or channels change. Both peers may send one at the same moment. Then the **polite** peer drops its own offer and answers the other one, and the impolite peer ignores the offer it got. + +Exactly one of two peers must be polite. Peers are impolite by default, so set `polite(true)` on one side, for example on the peer that joined second. When two impolite peers collide, neither gives way. An impolite peer that waited 5 seconds for an answer reports a `NegotiationException` that says to make one of them polite. @@ -150,23 +156,22 @@ Peer peer = rtc.createPeer(PeerOptions.builder() .build()); ``` -`DISCONNECTED` starts the grace period; `FAILED` skips it. Repeated failure -notifications during an attempt do not start parallel restarts. Events are -`WAITING`, `ATTEMPTING`, `RETRYING`, `RECOVERED` and `EXHAUSTED`, delivered on the -peer's event executor. `attempt()` counts attempts in the current outage, starting -at one; a recovery during the grace period reports zero. `cause()` gives the last -failure on retry or exhaustion. Successful signaling alone does not count as -recovery: the peer must report `CONNECTED`. That resets the budget, including -after exhaustion. Exhaustion leaves the peer open and creates no further automatic -attempts until it connects again. `whenConnected()` keeps its existing first-connection -semantics; observe `RECOVERED` for later recovery. - -Before an attempt, `iceServers(attempt -> completionStage)` can obtain fresh TURN -credentials asynchronously. It runs on the event executor and must return promptly; -the resulting list replaces **all** ICE servers, so include STUN servers that should -remain available. The library copies the supplied servers; failed, null or invalid -results fail the attempt. For example, with an application's asynchronous credential -service returning `CompletionStage>`: +- `DISCONNECTED` starts the grace period; `FAILED` skips it. +- Repeated failure notifications during an attempt do not start parallel restarts. +- The events are `WAITING`, `ATTEMPTING`, `RETRYING`, `RECOVERED` and `EXHAUSTED`, delivered on the peer's event executor. +- `attempt()` counts the attempts in the current outage, starting at one. A recovery during the grace period reports zero. +- `cause()` gives the last failure, on retry or exhaustion. +- Successful signaling alone does not count as recovery: the peer must report `CONNECTED`. That resets the budget, also after exhaustion. +- Exhaustion leaves the peer open, and creates no further automatic attempts until the peer connects again. +- `whenConnected()` keeps its meaning: the first connection. Observe `RECOVERED` for later recoveries. + +Before an attempt, `iceServers(attempt -> completionStage)` can obtain fresh TURN credentials asynchronously: + +- It runs on the event executor and must return promptly. +- The resulting list replaces **all** ICE servers, so include the STUN servers that should remain available. +- The library copies the servers it is given. A failed, null or invalid result fails the attempt. + +For example, with an application's asynchronous credential service that returns `CompletionStage>`: ```java RecoveryPolicy recovery = RecoveryPolicy.builder() @@ -179,25 +184,24 @@ RecoveryPolicy recovery = RecoveryPolicy.builder() .build(); ``` -ICE recovery requires working signaling. Reconnecting a WebSocket or refreshing -HTTP authorization is the application's separate responsibility. A policy does not -create a replacement peer or session. If the application chooses that fallback, it -must close the old session, attach handlers to the new peer, republish local tracks, -and handle new remote tracks and data channels explicitly; their old handles are -not transferred. +ICE recovery requires working signaling. Reconnecting a WebSocket or refreshing HTTP authorization is the application's separate responsibility. A policy does not create a replacement peer or session. + +If the application chooses that fallback, it must close the old session and attach handlers to the new peer. It must also republish local tracks, and handle new remote tracks and data channels explicitly: their old handles are not transferred. + +`Whip.to(endpoint).recovery(recovery)` and `Whep.from(endpoint).recovery(recovery)` use the same policy with the session's ICE `PATCH` exchange. This replaces their built-in restart rule for that session. -`Whip.to(endpoint).recovery(recovery)` and `Whep.from(endpoint).recovery(recovery)` -use the same policy with the session's ICE `PATCH` exchange. This replaces their -built-in restart rule for that session. Keep using `session.restartIce()` for a -manual HTTP restart; `session.peer().restartIce()` does not perform the HTTP -exchange. For a manually negotiated peer outside WHIP/WHEP, the application owns -recovery and its protocol-specific offer/answer exchange. +- For a manual HTTP restart, keep using `session.restartIce()`. `session.peer().restartIce()` does not perform the HTTP exchange. +- For a manually negotiated peer outside WHIP and WHEP, the application owns recovery and its protocol-specific offer and answer exchange. How attempts are timed and cancelled is described under [Details](#ice-recovery-timing). ## Manual Negotiation -SFUs and other servers dictate an offer and answer exchange of their own. A peer built with `Negotiation.MANUAL` leaves it to the application, through `peer.negotiator()`; `Whip` and `Whep` are built on it. Each step creates and applies a description in one go, and `Gathering.COMPLETE` waits until the description holds all local candidates, for servers that take no trickled candidates. A STUN or TURN server that cannot be reached, as on networks that block UDP, holds up the end of gathering for a long time, so `COMPLETE` waits at most for the peer's gathering timeout and then goes with the candidates it has. This WHEP client does by hand what `Whep` does: +SFUs and other servers dictate an offer and answer exchange of their own. A peer built with `Negotiation.MANUAL` leaves it to the application, through `peer.negotiator()`; `Whip` and `Whep` are built on it. Each step creates and applies a description in one go. + +`Gathering.COMPLETE` waits until the description holds all local candidates, for servers that take no trickled candidates. A STUN or TURN server that cannot be reached, as on networks that block UDP, holds up the end of gathering for a long time. `COMPLETE` therefore waits at most for the peer's gathering timeout, and then goes with the candidates it has. + +This WHEP client does by hand what `Whep` does: ```java Peer viewer = rtc.createPeer(PeerOptions.builder() @@ -219,7 +223,9 @@ viewer.negotiator().createOffer(Gathering.COMPLETE) | `createOffer()`, `createOffer(Gathering)` | Creates the local offer and applies it. | | `answer(offer)`, `answer(offer, Gathering)` | Applies the remote offer, then creates the answer and applies it. | | `applyAnswer(answer)` | Applies the remote answer to the local offer. | -| `addCandidate(candidate)` | Adds a remote candidate, or holds it until there is a remote description to add it to. The future completes once the candidate is added or held; a held candidate that WebRTC rejects later reaches the error handler. | +| `addCandidate(candidate)` | Adds a remote candidate, or holds it until there is a remote description to add it to. | + +The future of `addCandidate` completes once the candidate is added or held. A held candidate that WebRTC rejects later reaches the error handler. A manual peer with a `Signaling` still sends its local candidates through it. `peer.negotiator()` throws an `IllegalStateException` on a peer that negotiates automatically. @@ -227,13 +233,20 @@ A manual peer with a `Signaling` still sends its local candidates through it. `p ### Parsing Signal Messages -Anything else throws an `IllegalArgumentException` that says what is wrong and where, as does text longer than a MiB or nested deeper than 64 levels, so a message from the remote side cannot exhaust the stack. The JSON must be strict (RFC 8259): no raw line breaks or other control characters inside strings, no member name twice in one object, no numbers like `01`, `1.` or `1e999`. An `sdpMLineIndex` must be an integer from 0 to 65535; `1.9` is rejected rather than truncated to 1. +Anything else throws an `IllegalArgumentException` that says what is wrong and where. So does text longer than a MiB, or nested deeper than 64 levels, so that a message from the remote side cannot exhaust the stack. + +The JSON must be strict (RFC 8259): + +- no raw line breaks or other control characters inside strings, +- no member name twice in one object, +- no numbers like `01`, `1.` or `1e999`. + +An `sdpMLineIndex` must be an integer from 0 to 65535; `1.9` is rejected rather than truncated to 1. ### ICE Recovery Timing -The attempt deadline covers credentials, signaling and waiting for connectivity. -An HTTP restart already in flight finishes or reaches its own operation timeout -before another starts. Late credentials and completions after recovery or `close()` -are ignored; the credential service owns cancellation of its requests. Closing the -peer or runtime cancels scheduled retries, including one whose timer has fired but -whose control task has not run yet. Event notifications already queued may still arrive. +The attempt deadline covers credentials, signaling and waiting for connectivity. An HTTP restart already in flight finishes, or reaches its own operation timeout, before another starts. + +- Late credentials, and completions after recovery or `close()`, are ignored. Cancelling its requests is up to the credential service. +- Closing the peer or the runtime cancels scheduled retries, including one whose timer has fired but whose control task has not run yet. +- Event notifications already queued may still arrive. \ No newline at end of file From 1c5d8afce63433cdd55e2788d8532fa80ee3fb98 Mon Sep 17 00:00:00 2001 From: Alex Andres Date: Fri, 13 Nov 2026 01:42:06 +0100 Subject: [PATCH 09/37] docs: write the encoded transforms page in shorter sentences Same content: rules as lists, long paragraphs split. --- docs/guide/advanced/encoded-transforms.md | 41 ++++++++++++++--------- 1 file changed, 25 insertions(+), 16 deletions(-) diff --git a/docs/guide/advanced/encoded-transforms.md b/docs/guide/advanced/encoded-transforms.md index 6c4297c5..5080507f 100644 --- a/docs/guide/advanced/encoded-transforms.md +++ b/docs/guide/advanced/encoded-transforms.md @@ -1,17 +1,10 @@ # Encoded Transforms -This guide explains how to read and change the encoded frames of a call as they pass through an `RTCRtpSender` or an `RTCRtpReceiver`, the way [WebRTC Encoded Transforms](https://www.w3.org/TR/webrtc-encoded-transform/) (also known as insertable streams) do in a browser. It covers: +This guide explains how to read and change the encoded frames of a call as they pass through an `RTCRtpSender` or an `RTCRtpReceiver`. It works the way [WebRTC Encoded Transforms](https://www.w3.org/TR/webrtc-encoded-transform/), also known as insertable streams, do in a browser. -- Setting a transform on a sender or a receiver -- Reading and replacing a frame's payload -- Dropping frames -- What each frame tells about itself -- Asking for key frames -- What end-to-end encryption needs of a transform -- Threading and performance -- Sending frames that are already encoded through an injector +A sender's transform sees every frame after the encoder and before the packetizer. A receiver's transform sees every frame after the depacketizer and before the decoder. Everything between the two, whether that is the network, a TURN server or an SFU, only ever carries what the transform produced. -A sender's transform sees every frame after the encoder and before the packetizer; a receiver's sees every frame after the depacketizer and before the decoder. Everything between the two, whether that is the network, a TURN server or an SFU, only ever carries what the transform produced. That is what makes end-to-end encryption possible, and also frame metadata, watermarks, or analysis of the encoded stream. +That is what makes end-to-end encryption possible, and also frame metadata, watermarks, or analysis of the encoded stream. ## Setting a Transform @@ -39,7 +32,11 @@ for (RTCRtpReceiver receiver : peerConnection.getReceivers()) { Set the transform of a video sender before negotiating when you can. The first transform set on a running video sender restarts its send stream, which costs a key frame. Later changes, including removing it, cost nothing. ::: -The transform belongs to the native sender or receiver, not to the Java object: every `RTCRtpSender` instance for the same sender shares it, setting it through one replaces what was set through another, and disposing of an instance leaves it in place. +The transform belongs to the native sender or receiver, not to the Java object: + +- Every `RTCRtpSender` instance for the same sender shares it. +- Setting it through one instance replaces what was set through another. +- Disposing of an instance leaves it in place. ### With the Client API @@ -71,7 +68,9 @@ receiver.setTransform(frame -> { }); ``` -`setData` takes a `ByteBuffer` (its remaining bytes) or a `byte[]`, and the new payload may be larger or smaller than the old one. The payload is only copied into Java when `getData()` is called, so a transform that looks at the metadata alone costs no copy at all. Each transform thread reuses at most 256 KiB of direct scratch memory. Larger payloads use a temporary buffer sized to the frame, released for garbage collection when the transform returns. Workloads with frequent larger frames therefore allocate a buffer for each of them. +`setData` takes a `ByteBuffer` (its remaining bytes) or a `byte[]`, and the new payload may be larger or smaller than the old one. The payload is only copied into Java when `getData()` is called, so a transform that looks at the metadata alone costs no copy at all. + +Each transform thread reuses at most 256 KiB of direct scratch memory. Larger payloads use a temporary buffer sized to the frame, released for garbage collection when the transform returns. Workloads with frequent larger frames therefore allocate a buffer for each of them. A frame, and the buffer `getData()` returned, is valid only while `transform` runs and only on its thread. The buffer's memory is reused for the next frame, so copy what you need to keep. Reading or changing a frame after the transform returned throws an `IllegalStateException`; its metadata stays readable. @@ -105,10 +104,16 @@ Both do nothing for audio, and `generateKeyFrame()` nothing before the connectio ## End-to-End Encryption -[End-to-End Encryption](/guide/advanced/end-to-end-encryption) builds on these transforms: a `KeyProvider` makes the encryptors and decryptors, in the SFrame format of RFC 9605, and takes care of the keys of every participant, key rotation, replay protection and frames that fail. Use it rather than a transform of your own, which would have to settle all of that, and a frame format browsers can implement too. Two points matter for any encrypting transform that is written anyway: +[End-to-End Encryption](/guide/advanced/end-to-end-encryption) builds on these transforms. A `KeyProvider` makes the encryptors and decryptors, in the SFrame format of RFC 9605. It also takes care of the keys of every participant, key rotation, replay protection and frames that fail. + +Use it rather than a transform of your own, which would have to settle all of that, and a frame format that browsers can implement too. Two points matter for any encrypting transform that is written anyway: -- **Keep the codec header in the clear.** The packetizer and depacketizer read the start of some frames: a VP8 receiver finds key frames and the frame size in the first bytes, and the AV1 packetizer splits frames into their OBUs. Browsers leave 10 bytes of a VP8 key frame, 3 of a VP8 delta frame and 1 of an Opus frame unencrypted, and authenticate them as associated data instead. H.264 is split at its NAL unit start codes before it is sent, so encrypting it whole breaks the packetizer. -- **Make each frame self-contained.** Frames are lost, so each needs what it takes to be decrypted, such as its nonce, and a receiver must drop a frame that fails to authenticate rather than pass it on. +- **Keep the codec header in the clear.** The packetizer and depacketizer read the start of some frames: + - A VP8 receiver finds key frames and the frame size in the first bytes. + - The AV1 packetizer splits frames into their OBUs. + - H.264 is split at its NAL unit start codes before it is sent, so encrypting it whole breaks the packetizer. + + Browsers leave 10 bytes of a VP8 key frame, 3 of a VP8 delta frame and 1 of an Opus frame unencrypted, and authenticate them as associated data instead. A transform that protects what passes through it can keep observers of the encoded frames, such as a [`MediaRecorder`](/guide/media/media-recording), from seeing it in the clear by returning false from `allowsPlaintextObservers()`. @@ -148,6 +153,10 @@ RTCEncodedVideoFrameInjector injector = transceiver.getSender() injector.injectFrame(frameData, rtpTimestamp, keyFrame, width, height); ``` -The frames must be in the codec the sender negotiated, which `setCodecPreferences` on the transceiver can make sure of, and the RTP timestamp runs on the 90 kHz video clock. VP9 cannot be injected: WebRTC sends injected frames without the scalability structure a VP9 receiver needs with every key frame, so the receiver drops them. H.264, H.265 and AV1 frames arrive. An audio sender takes an `RTCEncodedAudioFrameInjector` the same way, whose frames carry the payload type of their codec, from the sender's `getParameters().codecs`. Do not inject audio through a factory that sends audio captured by its audio device module, as WebRTC feeds captured audio to every audio sender. +The frames must be in the codec the sender negotiated, which `setCodecPreferences` on the transceiver can make sure of. The RTP timestamp runs on the 90 kHz video clock. + +VP9 cannot be injected: WebRTC sends injected frames without the scalability structure that a VP9 receiver needs with every key frame, so the receiver drops them. H.264, H.265 and AV1 frames arrive. + +An audio sender takes an `RTCEncodedAudioFrameInjector` the same way. Its frames carry the payload type of their codec, from the sender's `getParameters().codecs`. Do not inject audio through a factory that sends audio captured by its audio device module: WebRTC feeds captured audio to every audio sender. Giving the sender a track through `replaceTrack` ends the injection, and `dispose()` releases the injector. The observer runs on a callback thread shared with other WebRTC observers, so it may call back into WebRTC but should not block. From df1a3c1ab0276d8c819f1a266c5a7ff757768a15 Mon Sep 17 00:00:00 2001 From: Alex Andres Date: Fri, 13 Nov 2026 01:42:38 +0100 Subject: [PATCH 10/37] docs: write the core API page in shorter sentences Same content: the rules of disposal as a list. --- docs/guide/core-api.md | 17 +++++++++++------ 1 file changed, 11 insertions(+), 6 deletions(-) diff --git a/docs/guide/core-api.md b/docs/guide/core-api.md index ec4cf57c..192ba14b 100644 --- a/docs/guide/core-api.md +++ b/docs/guide/core-api.md @@ -126,7 +126,7 @@ peerConnection.setLocalDescription(new SetSessionDescriptionObserver() { ### Adding Candidates and Knowing the Outcome -`addIceCandidate(candidate)` reports nothing and fails silently when the remote description is not set yet. The form that takes an `AddIceCandidateObserver` is queued behind any pending offer/answer operation, as the standard requires, so a candidate can be added right after `setRemoteDescription()` without waiting for it, and the observer says whether it was accepted: +`addIceCandidate(candidate)` reports nothing and fails silently when the remote description is not set yet. The form that takes an `AddIceCandidateObserver` is queued behind any pending offer/answer operation, as the standard requires. A candidate can therefore be added right after `setRemoteDescription()`, without waiting for it, and the observer says whether it was accepted: ```java peerConnection.addIceCandidate(candidate, new AddIceCandidateObserver() { @@ -225,7 +225,7 @@ RTCRtpSender audioSender = peerConnection.addTrack(audioTrack, streamIds); ### Receiving Tracks -A remote track arrives in `onTrack` with the transceiver that carries it. To learn when its media actually starts flowing, for example to show a video only once there is something to show, set an `RTCRtpReceiverObserver`; an `RTCRtpSenderObserver` does the same for the first packet a sender sends. Either is called right away when that has already happened: +A remote track arrives in `onTrack` with the transceiver that carries it. To learn when its media actually starts flowing, set an `RTCRtpReceiverObserver`, for example to show a video only once there is something to show. An `RTCRtpSenderObserver` does the same for the first packet a sender sends. Either is called right away when that has already happened: ```java @Override @@ -269,9 +269,12 @@ peerConnection.close(); factory.dispose(); ``` -A track may be disposed, and a listener removed, from within one of its listeners: the listeners the notification has not reached yet are then not called, and what goes is freed once the notification is over. An observer registered on one thread while the object is disposed on another is freed once and never left registered. A factory disposed while a closed connection is still being freed is released once that connection is. - -`dispose()` and `close()` may be called from any thread and more than once: only the first call frees anything, later ones do nothing. Once an object is disposed, its methods throw an `IllegalStateException` (a closed connection still reports its state). A method that another thread is running at that moment keeps the native object until it returns, and what the dispose frees is freed then. +- A track may be disposed, and a listener removed, from within one of its listeners. The listeners the notification has not reached yet are then not called, and what goes is freed once the notification is over. +- An observer registered on one thread while the object is disposed on another is freed once, and never left registered. +- A factory disposed while a closed connection is still being freed is released once that connection is. +- `dispose()` and `close()` may be called from any thread and more than once. Only the first call frees anything; later ones do nothing. +- Once an object is disposed, its methods throw an `IllegalStateException`. A closed connection still reports its state. +- A method that another thread is running at that moment keeps the native object until it returns. What the dispose frees is freed then. Apart from the media sources, which only have `dispose()`, objects with a `dispose()` method also have a `close()` that does the same. A try-with-resources statement therefore releases them, also when an exception leaves the block: @@ -297,4 +300,6 @@ The reports are warnings through `Logging`, which writes nothing unless the appl Logging.logToDebug(Logging.Severity.WARNING); ``` -Only objects that own a native object and have a `dispose()` or `close()` method are watched: not views such as the track of `RTCRtpSender.getTrack()`, which own nothing, and not reference-counted objects such as video frame buffers. The detector only reports; it never frees anything itself, since a native object may still be in use on a WebRTC thread. While it is off, which is the default, nothing is watched. +Only objects that own a native object and have a `dispose()` or `close()` method are watched. Not watched are views, such as the track of `RTCRtpSender.getTrack()`, which own nothing, and reference-counted objects, such as video frame buffers. + +The detector only reports. It never frees anything itself, since a native object may still be in use on a WebRTC thread. While it is off, which is the default, nothing is watched. From b038b1c4f86fef5e53db93c2928c958e4df010f8 Mon Sep 17 00:00:00 2001 From: Alex Andres Date: Fri, 13 Nov 2026 01:43:23 +0100 Subject: [PATCH 11/37] docs: write the threads page in shorter sentences Same content: the exceptions of the thread table as a list below it, and screens() and windows() named as asking the system on every call. --- docs/guide/client/threads.md | 34 ++++++++++++++++++++++++++-------- 1 file changed, 26 insertions(+), 8 deletions(-) diff --git a/docs/guide/client/threads.md b/docs/guide/client/threads.md index 180b02da..ee6ac96f 100644 --- a/docs/guide/client/threads.md +++ b/docs/guide/client/threads.md @@ -4,13 +4,25 @@ Every method of the client API may be called from any thread, including from ins | Code | Runs on | May it block? | |---|---|---| -| Your calls into the API | your thread | They don't block, with these exceptions. `WebRtc.createPeer`, the `LocalMedia` methods that make a track and `LocalTrack.setEnabled` and `LocalVideo.setContentHint` wait briefly for WebRTC. `camera()` and `screen()` also open their capture on your thread, which can take longer, so a UI thread should leave them to another thread. The first `media.devices()`, `screens()` and `windows()` ask the system on your thread. `WebRtc.close()` waits up to the close timeout. | -| Peer, track and channel handlers, future callbacks, `Signaling.send` | the peer's event thread | Yes, but that delays only the events of this peer. They must not wait for a future of this API, see below. | -| `onFrame` and `onAudio` handlers | a thread of each subscription | Better not. A slow video handler gets the newest frame next, and the frames in between are dropped; a slow audio handler loses the oldest audio beyond its buffer. | +| Your calls into the API | your thread | No, apart from the exceptions below the table. | +| Peer, track and channel handlers, future callbacks, `Signaling.send` | the peer's event thread | Yes, but that delays only the events of this peer. They must not [wait for a future](#don-t-wait-in-a-handler) of this API. | +| `onFrame` and `onAudio` handlers | a thread of each subscription | Better not. A slow video handler misses frames; a slow audio handler loses the oldest audio beyond its buffer. | | [Encoded transforms](/guide/client/transforms) | a thread of each sender and receiver | Better not. A slow transform delays only its own frames, and one that falls several seconds behind has frames dropped. | -| `LocalTrack.onEnded` and `Devices.onChange` handlers, callbacks of the futures of local tracks and devices | the event thread of the runtime's local media | Yes, but that delays the other events of local media. They must not wait for a future of this API either. A track is released before its `onEnded` handlers run, so they never hold up a release. | +| `LocalTrack.onEnded` and `Devices.onChange` handlers, callbacks of the futures of local tracks and devices | the event thread of the runtime's local media | Yes, but that delays the other events of local media. They must not wait for a future of this API either. | | WebRTC's own callbacks | WebRTC's threads | Not your code. The library only copies what they report and queues it. | +Calls into the API do not block, with these exceptions: + +- `WebRtc.createPeer`, the `LocalMedia` methods that make a track, `LocalTrack.setEnabled` and `LocalVideo.setContentHint` wait briefly for WebRTC. +- `camera()` and `screen()` also open their capture on your thread, which can take longer. A UI thread should leave them to another thread. +- The first `media.devices()`, and every `screens()` and `windows()`, ask the system on your thread. +- `WebRtc.close()` waits up to the close timeout. + +About the handlers in the table: + +- A slow video handler gets the newest frame next, and the frames in between are dropped. +- A track is released before its `onEnded` handlers run, so they never hold up a release. + The handlers of one peer never run at the same time, and they run in the order of the events. The handlers of different peers may run at the same time. With the default executor, the event thread is named after its peer, such as `webrtc-client-events-alice`, which shows up in thread dumps. An executor of your own that rejects work is described under [Details](#executors-that-reject-work). @@ -26,13 +38,17 @@ WebRtc rtc = WebRtc.builder() .build(); ``` -With a limit, the peers take turns, and so do the subscriptions: one that has work runs up to 64 handlers, for at most 10 ms, then goes to the back of the line. A slow handler thus delays the others but never starves them, and a peer's events still run one at a time and in order. The limit rejects nothing: events wait for their turn and are never dropped, so no future stays open, and once the runtime closes, the futures still open fail after the events queued before them. Media that waits is bounded by its subscription instead, by the `FrameDelivery` of a video handler and the audio buffer of an audio handler, which drop the oldest frames and audio when they are full. +With a limit, the peers take turns, and so do the subscriptions. One that has work runs up to 64 handlers, for at most 10 ms, and then goes to the back of the line. A slow handler thus delays the others but never starves them, and a peer's events still run one at a time and in order. -A limit has one cost: a handler that waits for what another peer's handler does, say on a latch, can wait forever once all threads are taken by such waits. Handlers that wait for something outside the API should hand the work to a thread of their own. +The limit rejects nothing: events wait for their turn and are never dropped, so no future stays open. Once the runtime closes, the futures still open fail after the events queued before them. + +Media that waits is bounded by its subscription instead: by the `FrameDelivery` of a video handler, and by the audio buffer of an audio handler. They drop the oldest frames and audio when they are full. + +A limit has one cost. A handler that waits for what another peer's handler does, say on a latch, can wait forever once all threads are taken by such waits. Handlers that wait for something outside the API should hand the work to a thread of their own. The control threads of the library, which call into WebRTC, are not limited. Their tasks wait for one another, such as a camera switch for every peer to take the new track. -The HTTP requests of `Whip` and `Whep` sessions take no thread while they wait, so a slow server holds up no peer: they go through Java's `HttpClient`, one of each runtime. +The HTTP requests of `Whip` and `Whep` sessions take no thread while they wait, so a slow server holds up no peer. They go through Java's `HttpClient`, one per runtime. `rtc.loadStats()` tells how much work waits and runs, at this moment and cheaply enough to be read every second: @@ -99,4 +115,6 @@ A UI thread used as event executor does not refuse: waiting on it for a future o ### Executors That Reject Work -An executor of your own that rejects work, such as a bounded pool that is full, loses the events it rejects, and the rejection goes to the error handler. The futures those events would have completed complete all the same, on the thread that found them rejected. Once the runtime is closed, every future of this API that is not done fails with `PeerClosedException`, including those of calls made after the close. +An executor of your own that rejects work, such as a bounded pool that is full, loses the events it rejects. The rejection goes to the error handler. + +The futures those events would have completed complete all the same, on the thread that found them rejected. Once the runtime is closed, every future of this API that is not done fails with `PeerClosedException`, including those of calls made after the close. From b6011b0bea9b3058a14ac694a2da8a060ed3ec59 Mon Sep 17 00:00:00 2001 From: Alex Andres Date: Fri, 13 Nov 2026 01:43:46 +0100 Subject: [PATCH 12/37] docs: write the receiving media page in shorter sentences Same content: the memory rules of the video views and the speech detection settings as lists. --- docs/guide/client/receiving-media.md | 42 +++++++++++++++++----------- 1 file changed, 26 insertions(+), 16 deletions(-) diff --git a/docs/guide/client/receiving-media.md b/docs/guide/client/receiving-media.md index 2eceedeb..a2bcd19a 100644 --- a/docs/guide/client/receiving-media.md +++ b/docs/guide/client/receiving-media.md @@ -22,7 +22,9 @@ A track ends when the remote peer stops sending it or the peer closes. ## Video Frames -Each `onFrame` subscription has a thread of its own, so a slow renderer holds up neither WebRTC's decoder nor the peer's other events. With `frameThreads(n)` the subscriptions share `n` threads and take turns, see [Limiting Threads](/guide/client/threads#limiting-threads). By default the handler always gets the newest frame, and the frames it had no time for are dropped. `FrameDelivery.queue(n)` keeps up to `n` frames instead: +Each `onFrame` subscription has a thread of its own, so a slow renderer holds up neither WebRTC's decoder nor the peer's other events. With `frameThreads(n)`, the subscriptions share `n` threads and take turns (see [Limiting Threads](/guide/client/threads#limiting-threads)). + +By default, the handler always gets the newest frame, and the frames it had no time for are dropped. `FrameDelivery.queue(n)` keeps up to `n` frames instead: ```java video.onFrame(FrameDelivery.queue(10), frame -> encoder.encode(frame)); @@ -96,9 +98,11 @@ The view converts frames on their own thread and paints at most once per JavaFX ## Audio -`onAudio` hands decoded audio to a handler in order, usually 10 ms of 16-bit PCM per `AudioChunk`. Audio reaches these handlers only while the runtime plays out, which a `HEADLESS` runtime always does and a `DEVICES` or `CUSTOM` runtime does to its speakers. `setVolume(double)` changes the playout gain of a track, from 0 to 10, where 1 plays it as received. +`onAudio` hands decoded audio to a handler in order, usually 10 ms of 16-bit PCM per `AudioChunk`. Audio reaches these handlers only while the runtime plays out. A `HEADLESS` runtime always does; a `DEVICES` or `CUSTOM` runtime plays out to its speakers. `setVolume(double)` changes the playout gain of a track, from 0 to 10, where 1 plays it as received. + +A handler that falls behind gets the audio of the last 5 seconds. The oldest audio beyond that is dropped, and counted in `rtc.loadStats().audio().dropped()`. `WebRtc.Builder.audioBuffer(Duration)` changes the buffer of every subscription, and `onAudio(Duration, handler)` that of one. -A handler that falls behind gets the audio of the last 5 seconds; the oldest audio beyond that is dropped and counted in `rtc.loadStats().audio().dropped()`. `WebRtc.Builder.audioBuffer(Duration)` changes the buffer of every subscription, and `onAudio(Duration, handler)` that of one. A speech recognizer that must stay live keeps a short buffer; a recorder that can catch up after a stall keeps a long one, at about 190 KiB per second of 48 kHz stereo: +A speech recognizer that must stay live keeps a short buffer. A recorder that can catch up after a stall keeps a long one, at about 190 KiB per second of 48 kHz stereo: ```java track.asAudio().onAudio(Duration.ofMillis(200), chunk -> recognizer.feed(chunk.samples())); @@ -113,7 +117,12 @@ Subscription speaking = microphone.onVoiceActivity( active -> System.out.println(active ? "speaking" : "quiet")); ``` -`VoiceActivityConfig` sets the threshold, the voice probability from 0 to 1 at which audio counts as speech (0.5 by default), and the hangover, how long speech goes on once the audio fell below it (300 ms by default), which bridges the pauses between words. Which audio is analyzed is described under [Details](#speech-detection). +`VoiceActivityConfig` sets two values: + +- the threshold: the voice probability, from 0 to 1, at which audio counts as speech (0.5 by default), +- the hangover: how long speech goes on once the audio fell below the threshold (300 ms by default), which bridges the pauses between words. + +Which audio is analyzed is described under [Details](#speech-detection). ## Receiving Without Sending @@ -146,18 +155,19 @@ Closing the peer, or removing a sender, during a recording ends that track in th ### Video Views and Memory -Both views support resolutions such as 8K without a fixed pixel limit. They reject -nonpositive dimensions and frames whose four-byte pixel data cannot fit in a Java -buffer, using overflow-safe arithmetic. Conversion buffers allocate on demand, -grow geometrically when necessary, and retain their capacity across smaller frames -and `clear()`. Swing reuses one direct buffer; JavaFX reuses up to two. -JavaFX reserves a buffer only while copying it into the displayed image; a waiting -frame may be replaced by a newer frame to avoid building a queue. Buffer -memory is released when the view becomes unreachable. Memory usage follows the -largest frames received and includes displayed images, conversion buffers, and -replacement allocations awaiting garbage collection. Snapshots, input frames and -toolkit rendering resources use additional memory. +Both views support resolutions such as 8K, without a fixed pixel limit. They reject nonpositive dimensions, and frames whose four-byte pixel data cannot fit in a Java buffer, using overflow-safe arithmetic. + +- Conversion buffers are allocated on demand, grow geometrically when necessary, and keep their capacity across smaller frames and `clear()`. +- Swing reuses one direct buffer; JavaFX reuses up to two. +- JavaFX reserves a buffer only while copying it into the displayed image. A waiting frame may be replaced by a newer one, so that no queue builds up. +- Buffer memory is released when the view becomes unreachable. + +Memory use follows the largest frames received. It includes displayed images, conversion buffers, and replacement allocations awaiting garbage collection. Snapshots, input frames and toolkit rendering resources use additional memory. ### Speech Detection -Pushed audio is analyzed as pushed. A microphone's audio is analyzed after audio processing, with echo cancellation, noise suppression and gain control applied as configured, so the remote peer's voice from the speakers does not count as this side speaking. It flows while WebRTC records, which it does while a peer sends a microphone track. Every microphone track of a runtime hears the same, as WebRTC records one microphone for all of them. +Pushed audio is analyzed as pushed. + +A microphone's audio is analyzed after audio processing, with echo cancellation, noise suppression and gain control applied as configured. The remote peer's voice from the speakers therefore does not count as this side speaking. + +The microphone's audio flows while WebRTC records, which it does while a peer sends a microphone track. Every microphone track of a runtime hears the same, as WebRTC records one microphone for all of them. \ No newline at end of file From c373e40e3765be226a476700993d86c17a304e55 Mon Sep 17 00:00:00 2001 From: Alex Andres Date: Fri, 13 Nov 2026 01:44:02 +0100 Subject: [PATCH 13/37] docs: write the client transforms page in shorter sentences Same content, long paragraphs split. --- docs/guide/client/transforms.md | 22 ++++++++++++++++------ 1 file changed, 16 insertions(+), 6 deletions(-) diff --git a/docs/guide/client/transforms.md b/docs/guide/client/transforms.md index 39b5f594..ca373124 100644 --- a/docs/guide/client/transforms.md +++ b/docs/guide/client/transforms.md @@ -1,6 +1,6 @@ # Transforms and Encryption -An [encoded transform](/guide/advanced/encoded-transforms) sees every encoded frame between the encoder and the network, or between the network and the decoder, which is where end-to-end encryption goes. The transforms of the peer options apply to every sender and every remote track of the peer: +An [encoded transform](/guide/advanced/encoded-transforms) sees every encoded frame between the encoder and the network, or between the network and the decoder. That is where end-to-end encryption goes. The transforms of the peer options apply to every sender and every remote track of the peer: ```java Peer peer = rtc.createPeer(PeerOptions.builder() @@ -10,11 +10,15 @@ Peer peer = rtc.createPeer(PeerOptions.builder() .build()); ``` -The peer sets the send transform on each sender as its track is added, before negotiation, so no frame leaves without it and the encoder needs no restart. It sets the receive transform on each receiver before `onTrack` hands out the track, so the transform is in place before the first frame arrives. What happens when it cannot be installed is described under [Details](#installing-the-receive-transform). The builders of [WHIP and WHEP](/guide/client/whip-whep) take them too, `Whip`'s as `sendTransform(...)`, `Whep`'s as `receiveTransform(...)`. +The peer sets the send transform on each sender as its track is added, before negotiation. No frame leaves without it, and the encoder needs no restart. + +The peer sets the receive transform on each receiver before `onTrack` hands out the track, so the transform is in place before the first frame arrives. What happens when it cannot be installed is described under [Details](#installing-the-receive-transform). + +The builders of [WHIP and WHEP](/guide/client/whip-whep) take transforms too: `Whip`'s as `sendTransform(...)`, `Whep`'s as `receiveTransform(...)`. ## End-to-End Encryption -For end-to-end encryption, `e2ee(keys)` takes the place of both transforms: every sender gets an encryptor and every remote track a decryptor of the [key provider](/guide/advanced/end-to-end-encryption), which also takes care of key rotation, replay protection, frames that fail and key frame requests: +For end-to-end encryption, `e2ee(keys)` takes the place of both transforms. Every sender gets an encryptor and every remote track a decryptor of the [key provider](/guide/advanced/end-to-end-encryption). The key provider also takes care of key rotation, replay protection, frames that fail and key frame requests: ```java PeerOptions.builder() @@ -32,7 +36,9 @@ videoSender.transform(frame -> addMetadata(frame)); remoteTrack.transform(null); // frames reach the decoder unchanged ``` -The first transform set on a video sender that already sends restarts its encoder, which costs a key frame. A transform set on a remote track misses the frames that arrive before its future completes, so a transform that every frame needs, such as decryption, belongs in the peer options. A recording of a sender holds its frames before the transform, a recording of a remote track after it. +- The first transform set on a video sender that already sends restarts its encoder, which costs a key frame. +- A transform set on a remote track misses the frames that arrive before its future completes. A transform that every frame needs, such as decryption, therefore belongs in the peer options. +- A recording of a sender holds its frames before the transform, a recording of a remote track after it. ## Key Frames @@ -47,10 +53,14 @@ remoteTrack.asVideo().requestKeyFrame(); // asks the remote sender for one ## Threads and Failures -Transforms run on a thread of each sender and receiver, not on the peer's event thread. A frame a transform throws on is dropped rather than passed on, so a failed encryption never sends a frame in the clear. The first such failure of each sender or receiver goes to the peer's `onError` handlers. The [`EncryptedRecordingExample`](https://github.com/devopvoid/webrtc-java/blob/main/webrtc-examples/src/main/java/dev/onvoid/webrtc/examples/EncryptedRecordingExample.java) encrypts a call with `e2ee(keys)` and records it. +Transforms run on a thread of each sender and receiver, not on the peer's event thread. + +A frame that a transform throws on is dropped rather than passed on, so a failed encryption never sends a frame in the clear. The first such failure of each sender or receiver goes to the peer's `onError` handlers. + +The [`EncryptedRecordingExample`](https://github.com/devopvoid/webrtc-java/blob/main/webrtc-examples/src/main/java/dev/onvoid/webrtc/examples/EncryptedRecordingExample.java) encrypts a call with `e2ee(keys)` and records it. ## Details ### Installing the Receive Transform -If installing the receive transform fails, the peer stops that track's transceiver and reports the failure through `onError`; the track is neither listed in `remoteTracks()` nor delivered through `onTrack`. If the receive path cannot be stopped, the peer closes. +If installing the receive transform fails, the peer stops that track's transceiver and reports the failure through `onError`. The track is neither listed in `remoteTracks()` nor delivered through `onTrack`. If the receive path cannot be stopped, the peer closes. \ No newline at end of file From b033a35a99ac4b95263cfed81ea4e20f23484b5e Mon Sep 17 00:00:00 2001 From: Alex Andres Date: Fri, 13 Nov 2026 01:44:25 +0100 Subject: [PATCH 14/37] docs: write the client API overview in shorter sentences Same content, long sentences split. --- docs/guide/client-api.md | 14 ++++++++++---- 1 file changed, 10 insertions(+), 4 deletions(-) diff --git a/docs/guide/client-api.md b/docs/guide/client-api.md index 14225676..087f5229 100644 --- a/docs/guide/client-api.md +++ b/docs/guide/client-api.md @@ -1,6 +1,6 @@ # Client API -The client API in `dev.onvoid.webrtc.client` is a layer over the rest of the library that takes care of the parts most applications get wrong: threads, the offer and answer exchange, and the lifetime of native objects. A whole video call fits on one screen: +The client API in `dev.onvoid.webrtc.client` is a layer over the rest of the library. It takes care of the parts most applications get wrong: threads, the offer and answer exchange, and the lifetime of native objects. A whole video call fits on one screen: ```java import dev.onvoid.webrtc.client.*; @@ -128,7 +128,7 @@ Failures arrive as exceptions, thrown or as the cause of a failed future, all of | `MediaException` | A device is missing, busy, refused access, or already open with other options. | | `ConfigurationException` | A call does not fit how the runtime was built, or WebRTC refused a configuration. | | `BlockingCallException` | A future was waited on from an event thread. | -| `HttpStatusException` | A WHIP or WHEP endpoint answered with an error status, which `status()` gives, with the `title()`, `detail()` and `problemType()` of the problem details it sent, if any. | +| `HttpStatusException` | A WHIP or WHEP endpoint answered with an error status: `status()`, and the `title()`, `detail()` and `problemType()` of its problem details, if any. | Failures that no caller waits for, such as a remote description rejected during automatic negotiation, go to the peer's `onError` handlers: @@ -168,8 +168,14 @@ mvn exec:java -D"exec.mainClass=dev.onvoid.webrtc.examples.ClientApiExample" -D" ### Timeouts -Every timeout of the client API, the operation and close timeouts as well as the [gathering timeout](/guide/client/peers#manual-negotiation) and the timeouts of [`Whip` and `Whep`](/guide/client/whip-whep), must be positive; none of them can be switched off. A timeout counts in milliseconds, rounded up, so one under a millisecond waits a millisecond. One too long for milliseconds to count waits as long as they can count, about 292 million years. +Every timeout of the client API must be positive; none of them can be switched off. This holds for the operation and close timeouts, the [gathering timeout](/guide/client/peers#manual-negotiation), and the timeouts of [`Whip` and `Whep`](/guide/client/whip-whep). + +A timeout counts in milliseconds, rounded up, so one under a millisecond waits a millisecond. One too long for milliseconds to count waits as long as they can count, about 292 million years. ### A Close That Fails -When a peer or track does not close within the close timeout, or a track or its source could not be released, the runtime leaves its factory undisposed rather than free what WebRTC's threads may still use. A track's `closeAsync()` fails in that case too. `WebRtc.closeAsync()` then fails with a `WebRtcException` naming the cause, and the error handler hears of it as well, which is how `close()` reports it. +A peer or track may not close within the close timeout, or a track or its source may not be released. The runtime then leaves its factory undisposed, rather than free what WebRTC's threads may still use. + +- A track's `closeAsync()` fails in that case too. +- `WebRtc.closeAsync()` fails with a `WebRtcException` naming the cause. +- The error handler hears of it as well, which is how `close()` reports it. From 4f6049fd379f66d26d05f3e0ed8b554ce8f5ee83 Mon Sep 17 00:00:00 2001 From: Alex Andres Date: Fri, 13 Nov 2026 01:48:00 +0100 Subject: [PATCH 15/37] docs: drop the outdated warning on injected audio Since #571 every audio sender gets the audio of its own track, so captured audio no longer reaches an injecting sender. --- docs/guide/advanced/encoded-transforms.md | 2 +- webrtc/src/main/java/dev/onvoid/webrtc/RTCRtpSender.java | 4 ---- 2 files changed, 1 insertion(+), 5 deletions(-) diff --git a/docs/guide/advanced/encoded-transforms.md b/docs/guide/advanced/encoded-transforms.md index 5080507f..f22d87ef 100644 --- a/docs/guide/advanced/encoded-transforms.md +++ b/docs/guide/advanced/encoded-transforms.md @@ -157,6 +157,6 @@ The frames must be in the codec the sender negotiated, which `setCodecPreference VP9 cannot be injected: WebRTC sends injected frames without the scalability structure that a VP9 receiver needs with every key frame, so the receiver drops them. H.264, H.265 and AV1 frames arrive. -An audio sender takes an `RTCEncodedAudioFrameInjector` the same way. Its frames carry the payload type of their codec, from the sender's `getParameters().codecs`. Do not inject audio through a factory that sends audio captured by its audio device module: WebRTC feeds captured audio to every audio sender. +An audio sender takes an `RTCEncodedAudioFrameInjector` the same way. Its frames carry the payload type of their codec, from the sender's `getParameters().codecs`. Giving the sender a track through `replaceTrack` ends the injection, and `dispose()` releases the injector. The observer runs on a callback thread shared with other WebRTC observers, so it may call back into WebRTC but should not block. diff --git a/webrtc/src/main/java/dev/onvoid/webrtc/RTCRtpSender.java b/webrtc/src/main/java/dev/onvoid/webrtc/RTCRtpSender.java index 4e8aa89f..b1612132 100644 --- a/webrtc/src/main/java/dev/onvoid/webrtc/RTCRtpSender.java +++ b/webrtc/src/main/java/dev/onvoid/webrtc/RTCRtpSender.java @@ -232,10 +232,6 @@ public RTCEncodedVideoFrameInjector createEncodedVideoFrameInjector( /** * Creates an injector that sends audio frames that are already encoded * on this sender, in place of a track. - *

- * Do not inject audio through a factory that sends audio captured by its - * audio device module: WebRTC passes captured audio to every audio sender - * that sends, and it would drive the injecting sender as well. * * @param observer The observer for the sender's target bitrate changes, * or {@code null} for none. From bc79949bcfd217ff86ab347dc85e3a535221a215 Mon Sep 17 00:00:00 2001 From: Alex Andres Date: Fri, 13 Nov 2026 01:50:44 +0100 Subject: [PATCH 16/37] docs: write the system audio source page in shorter sentences Same content: the client API first, the core API after it, and the rules of starting, stopping and feedback as lists. --- docs/guide/audio/system-audio-source.md | 145 +++++++++++------------- 1 file changed, 69 insertions(+), 76 deletions(-) diff --git a/docs/guide/audio/system-audio-source.md b/docs/guide/audio/system-audio-source.md index f6dc45dd..01aead42 100644 --- a/docs/guide/audio/system-audio-source.md +++ b/docs/guide/audio/system-audio-source.md @@ -1,23 +1,36 @@ # System Audio Source -`SystemAudioSource` captures audio playing on the system output. It is a -`CustomAudioSource` and can be passed directly to `PeerConnectionFactory.createAudioTrack`. -It sends 16-bit, 48 kHz stereo PCM in 10 ms chunks, independently of screen or -window video capture. Capturing a window does **not** restrict audio to that window. +`SystemAudioSource` captures the audio playing on the system output. It sends 16-bit, 48 kHz stereo PCM in 10 ms chunks, independently of screen or window video capture. Capturing a window does **not** restrict the audio to that window. -## Platform support +`SystemAudioSource` is a `CustomAudioSource`. It can be passed directly to `PeerConnectionFactory.createAudioTrack`, or to `audioTrack()` of the client API. -| Platform | Capture backend | Requirements and scope | -| --- | --- | --- | -| Windows | WASAPI shared-mode loopback | Captures the default multimedia output selected at start, including audio from this process. | -| Linux | PulseAudio output monitor | Requires `libpulse.so.0` and PulseAudio or PipeWire with `pipewire-pulse`. Captures the default sink selected at start, including audio from this process. A native PipeWire portal audio backend is not provided. | -| macOS 13+ | ScreenCaptureKit | Requires screen/audio recording permission and a shareable display. Captures system audio excluding this process. Build with a macOS 13+ SDK. | +## Platform Support -No shell commands, virtual audio driver, or FFmpeg installation are required. -There is no application/process selection API in this version. Protected content -and platform recording restrictions still apply. +| Platform | Capture backend | Captures | Needs | +| --- | --- | --- | --- | +| Windows | WASAPI shared-mode loopback | The default multimedia output selected at start, including audio from this process. | | +| Linux | PulseAudio output monitor | The default sink selected at start, including audio from this process. | `libpulse.so.0`, and PulseAudio or PipeWire with `pipewire-pulse`. | +| macOS 13+ | ScreenCaptureKit | System audio, excluding this process. | Screen and audio recording permission, a shareable display, and a build with a macOS 13+ SDK. | -## Send alongside a screen track +- A native PipeWire portal audio backend is not provided. +- No shell commands, virtual audio driver, or FFmpeg installation are required. +- There is no application or process selection API in this version. +- Protected content and platform recording restrictions still apply. + +## Send Alongside a Screen Track + +With the [Client API](/guide/client-api), pass the source to `audioTrack()`: + +```java +SystemAudioSource systemOutput = new SystemAudioSource(); +LocalAudio systemAudio = rtc.media().audioTrack("system-audio", systemOutput); +peer.addTrack(systemAudio, "presentation"); +systemOutput.start(); +``` + +The source stays yours. Close it only after the track, or after the runtime that closes the track. + +### Using the Core API Use your existing factory and peer connection, before negotiation: @@ -35,14 +48,7 @@ RTCRtpSender audioSender = peerConnection.addTrack( systemAudio.start(); ``` -`start()` waits for capture to open and throws `IllegalStateException` on startup -failure, with the platform error as its cause. Call it on a background thread, -particularly when macOS may show a permission prompt. Platform startup waits are -bounded; permission dialogs can remain open after a timeout. A late macOS grant -will not leave this timed-out capture recording in the background. Grant permission -in System Settings and retry, restarting the application if the OS requires it. - -When the call ends, stop recording before releasing tracks and the source: +When the call ends, stop recording before releasing the tracks and the source: ```java systemAudio.stop(); @@ -52,51 +58,44 @@ audioTrack.dispose(); systemAudio.dispose(); // or close(), also supported by try-with-resources ``` -With the [Client API](/guide/client-api), pass the source to `audioTrack()`. It -stays yours, so close it only after the track, or the runtime that closes it: +## Starting and Stopping -```java -SystemAudioSource systemOutput = new SystemAudioSource(); -LocalAudio systemAudio = rtc.media().audioTrack("system-audio", systemOutput); -peer.addTrack(systemAudio, "presentation"); -systemOutput.start(); -``` +`start()` waits for capture to open. If startup fails, it throws an `IllegalStateException`, with the platform error as its cause. + +- Call `start()` on a background thread, particularly when macOS may show a permission prompt. +- Platform startup waits are bounded, but permission dialogs can remain open after a timeout. A late macOS grant will not leave this timed-out capture recording in the background. +- Grant permission in System Settings and retry, restarting the application if the OS requires it. + +Use `try/finally` in application code, so that these resources are released on errors too. + +- `stop()` and `dispose()` are idempotent. +- `stop()` keeps the source and its tracks usable for `start()` again; disposal is permanent. +- A stop or disposal from a local audio sink requests shutdown without joining its own callback thread. Do not restart from an audio callback. +- The inherited `pushAudio` methods throw `IllegalStateException`. Push other audio into a separate `CustomAudioSource`. -Use `try/finally` in application code to release these resources on errors too. -`stop()` and `dispose()` are idempotent. `stop()` keeps the source and its tracks -usable for `start()` again; disposal is permanent. A stop or disposal from a local -audio sink requests shutdown without joining its own callback thread. Do not -restart from an audio callback. The inherited `pushAudio` methods throw -`IllegalStateException`: push other audio into a separate `CustomAudioSource`. +## Device and Permission Changes -## Device and permission changes +Observe `isRecording()` and `getFailure()` from your application's control thread. -Observe `isRecording()` and `getFailure()` from your application control thread. -A runtime capture failure stops recording and retains its cause in `getFailure()`; -starting again clears that failure. A stopped source does not end its tracks, so -it can resume on the same sender. It does not automatically reopen a device or -silently change recording permissions. +- A runtime capture failure stops recording, and keeps its cause in `getFailure()`. Starting again clears that failure. +- A stopped source does not end its tracks, so it can resume on the same sender. +- The source does not reopen a device by itself, and does not silently change recording permissions. -For an output change, call `stop()`, select the new default output through the OS, -and call `start()`. A change of the default output alone does not move a running -capture: Windows and Linux bind the output chosen at startup. Device loss or -permission revocation may only become observable when the platform reports it. +A change of the default output alone does not move a running capture: Windows and Linux bind the output chosen at startup. For an output change, call `stop()`, select the new default output through the OS, and call `start()`. -## Synchronization and latency +Device loss or permission revocation may only become observable when the platform reports it. -Capture timestamps use WebRTC's monotonic clock, the same clock as -`SyncClock.currentTimeUs()` and desktop video. WASAPI's packet clock and -ScreenCaptureKit's presentation clock are translated into that clock; PulseAudio -uses its reported capture latency. This preserves the capture time when audio is -buffered before delivery. Queueing is bounded to 100 ms; older chunks are dropped -if delivery stalls. Audio is paced by the OS rather than an independent Java timer. -This does not promise sample-accurate lip synchronization or compensate for all -screen capture, output-device, and receiver latency. +## Synchronization and Latency -## Microphone and feedback +Capture timestamps use WebRTC's monotonic clock, the same clock as `SyncClock.currentTimeUs()` and desktop video. WASAPI's packet clock and ScreenCaptureKit's presentation clock are translated into that clock; PulseAudio uses its reported capture latency. This preserves the capture time when audio is buffered before delivery. -Send the microphone as a second track of the same factory and connection. Each -sender gets the audio of its own track, so neither track hears the other: +- Queueing is bounded to 100 ms. Older chunks are dropped if delivery stalls. +- Audio is paced by the OS, not by an independent Java timer. +- This does not promise sample-accurate lip synchronization. It does not compensate for all screen capture, output device, and receiver latency. + +## Microphone and Feedback + +Send the microphone as a second track of the same factory and connection. Each sender gets the audio of its own track, so neither track hears the other: ```java AudioTrackSource microphone = factory.createAudioSource(new AudioOptions()); @@ -104,24 +103,18 @@ AudioTrack microphoneTrack = factory.createAudioTrack("microphone", microphone); peerConnection.addTrack(microphoneTrack, Collections.singletonList("presentation")); ``` -The microphone track gets the factory's echo cancellation, noise suppression and -gain control as its `AudioOptions` ask for; the system audio track gets none, and -the two are not mixed. Echo cancellation removes what this application plays, not -what other applications play, so system audio that reaches the microphone through -the speakers is sent twice: use headphones to prevent that. -On Windows and Linux, route received call audio to another output or to a virtual -output that is not monitored. Headphones alone do not prevent a loopback source -from capturing remote audio played on the monitored output. On macOS this process's -playout is excluded by ScreenCaptureKit; remote audio played in another process -can still be captured. - -Run `SystemAudioExample` from `webrtc-examples` for a capture-only example that -counts received PCM frames without playing them back, and -[`ScreenShareAudioExample`](/guide/examples#screen-share-with-audio) to send a -screen, the microphone and the system audio over one peer. The opt-in -`SystemAudioSourceIntegrationTest` (`-Dtest.systemAudio=true`) needs audio playing -on the default output and recording permission; lifecycle tests use deterministic -backends without hardware. +The microphone track gets the factory's echo cancellation, noise suppression and gain control, as its `AudioOptions` ask for. The system audio track gets none, and the two are not mixed. + +Echo cancellation removes what this application plays, not what other applications play. System audio that reaches the microphone through the speakers is therefore sent twice; use headphones to prevent that. + +- On Windows and Linux, route received call audio to another output, or to a virtual output that is not monitored. Headphones alone do not prevent a loopback source from capturing remote audio played on the monitored output. +- On macOS, ScreenCaptureKit excludes this process's playout. Remote audio played in another process can still be captured. + +## Examples and Tests + +- `SystemAudioExample` from `webrtc-examples` captures only, and counts received PCM frames without playing them back. +- [`ScreenShareAudioExample`](/guide/examples#screen-share-with-audio) sends a screen, the microphone and the system audio over one peer. +- The opt-in `SystemAudioSourceIntegrationTest` (`-Dtest.systemAudio=true`) needs audio playing on the default output, and recording permission. The lifecycle tests use deterministic backends without hardware. The platform bindings follow [Microsoft's loopback capture API](https://learn.microsoft.com/en-us/windows/win32/coreaudio/loopback-recording), [PulseAudio's recording stream API](https://www.freedesktop.org/software/pulseaudio/doxygen/stream_8h.html), From 552a99bac1baf5ada159567ffdf77877445cbe3e Mon Sep 17 00:00:00 2001 From: Alex Andres Date: Fri, 13 Nov 2026 01:51:14 +0100 Subject: [PATCH 17/37] docs: write the media constraints page in shorter sentences Same content: content hints and layers as lists. --- docs/guide/media/constraints.md | 16 ++++++++++++---- 1 file changed, 12 insertions(+), 4 deletions(-) diff --git a/docs/guide/media/constraints.md b/docs/guide/media/constraints.md index aa18000c..4c00013f 100644 --- a/docs/guide/media/constraints.md +++ b/docs/guide/media/constraints.md @@ -109,7 +109,7 @@ if (parameters.encodings != null && !parameters.encodings.isEmpty()) { } ``` -The `scaleResolutionDownBy` parameter specifies how much to scale down the video in each dimension. For example, a value of 2.0 means the video will be scaled down by a factor of 2 in both width and height, resulting in a video that is 1/4 the size of the original. +The `scaleResolutionDownBy` parameter specifies how much to scale down the video in each dimension. For example, a value of 2.0 halves both the width and the height, so the video is 1/4 the size of the original. ::: info Note that these constraints are applied without requiring SDP renegotiation, making them suitable for dynamic adaptation to changing network conditions. @@ -155,7 +155,10 @@ videoTrack.setContentHint(VideoContentHint.TEXT); // keep the resolution videoTrack.setContentHint(VideoContentHint.MOTION); // keep the frame rate ``` -`MOTION` keeps the frame rate, `DETAIL` and `TEXT` keep the resolution, and `DEFAULT` leaves it to the source, which keeps the resolution of screen shares. `DETAIL` and `TEXT` also put the encoder into its screen content mode, which VP8, VP9, H.264 (OpenH264) and AV1 code differently; WebRTC treats `TEXT` as `DETAIL` for now. The senders have taken the hint once `setContentHint` returns. It cannot be set from within the callback of a video sink, where it throws an `IllegalStateException`. +- `MOTION` keeps the frame rate. `DETAIL` and `TEXT` keep the resolution. `DEFAULT` leaves it to the source, which keeps the resolution of screen shares. +- `DETAIL` and `TEXT` also put the encoder into its screen content mode, which VP8, VP9, H.264 (OpenH264) and AV1 code differently. WebRTC treats `TEXT` as `DETAIL` for now. +- The senders have taken the hint once `setContentHint` returns. +- The hint cannot be set from within the callback of a video sink; there it throws an `IllegalStateException`. ## Simulcast @@ -188,7 +191,12 @@ The `rid`s cannot be changed afterwards; the other parameters of each encoding c ## Scalability Modes (SVC) -Instead of separate encodings, one encoding can carry layers, which a server can drop to lower the rate: temporal layers (fewer frames per second) and, with VP9 and AV1, spatial layers (lower resolutions). `scalabilityMode` takes the mode names of the [WebRTC SVC specification](https://www.w3.org/TR/webrtc-svc/), e.g. `L1T3` for three temporal layers, or `L3T3_KEY` for three spatial layers with three temporal layers each: +Instead of separate encodings, one encoding can carry layers, which a server can drop to lower the rate: + +- temporal layers, with fewer frames per second, +- with VP9 and AV1, spatial layers, with lower resolutions. + +`scalabilityMode` takes the mode names of the [WebRTC SVC specification](https://www.w3.org/TR/webrtc-svc/), for example `L1T3` for three temporal layers, or `L3T3_KEY` for three spatial layers with three temporal layers each: ```java RTCRtpSendParameters parameters = sender.getParameters(); @@ -200,7 +208,7 @@ The codec has to support the mode; `setParameters()` throws otherwise. `getScala ## Priorities -`bitratePriority` sets the share of the available bitrate a sender gets relative to the other senders of the peer connection (default 1.0), and `networkPriority` how its packets are marked on the network (DSCP), where the network honors it. Both apply to the whole sender, so only the first encoding may set them: +`bitratePriority` sets the share of the available bitrate a sender gets, relative to the other senders of the peer connection (default 1.0). `networkPriority` sets how its packets are marked on the network (DSCP), where the network honors it. Both apply to the whole sender, so only the first encoding may set them: ```java RTCRtpSendParameters parameters = sender.getParameters(); From dad4e12526cc57c3becf4c9b445118ddc9b74d89 Mon Sep 17 00:00:00 2001 From: Alex Andres Date: Fri, 13 Nov 2026 01:51:23 +0100 Subject: [PATCH 18/37] docs: write the custom video source page in shorter sentences Same content, rules about buffers as a list. --- docs/guide/video/custom-video-source.md | 15 ++++++++++++--- 1 file changed, 12 insertions(+), 3 deletions(-) diff --git a/docs/guide/video/custom-video-source.md b/docs/guide/video/custom-video-source.md index 18178dc7..e88f6fce 100644 --- a/docs/guide/video/custom-video-source.md +++ b/docs/guide/video/custom-video-source.md @@ -73,7 +73,13 @@ videoSource.pushFrame(frame); frame.release(); ``` -A frame can also carry a `VideoFrameBuffer` of your own. The source takes its pixels from `toI420()`: a `NativeI420Buffer` returned there is used as it is, and any other `I420Buffer` is copied, so its planes have to be direct byte buffers that are as large as their strides and the height require. A buffer that does not fit its description is rejected with a `RuntimeException`. `VideoFrame.copy()` works the same way, and always returns a frame with a `NativeI420Buffer`. +A frame can also carry a `VideoFrameBuffer` of your own. The source takes its pixels from `toI420()`: + +- A `NativeI420Buffer` returned there is used as it is. +- Any other `I420Buffer` is copied. Its planes have to be direct byte buffers, as large as their strides and the height require. +- A buffer that does not fit its description is rejected with a `RuntimeException`. + +`VideoFrame.copy()` works the same way, and always returns a frame with a `NativeI420Buffer`. ## Video Format Considerations @@ -82,7 +88,8 @@ When pushing video frames, you need to consider the following: ### Resolution - Common resolutions: 320x240, 640x480, 1280x720, 1920x1080 - Higher resolutions require more bandwidth and processing power -- Push frames at their full size: when the bitrate cannot carry them, the source scales them down (and, if asked, drops some) before they reach the encoder, the way a camera source does, and scales back up once the bandwidth allows. Local sinks on the track see the adapted frames too. +- Push frames at their full size. When the bitrate cannot carry them, the source scales them down before they reach the encoder, and, if asked, drops some. It does so the way a camera source does, and scales back up once the bandwidth allows. +- Local sinks on the track see the adapted frames too. ### Frame Rate - Common frame rates: 15, 24, 30, 60 fps @@ -149,7 +156,9 @@ public class VideoStreamer { ### Keeping the Source's Own Timing -`pushFrame(VideoFrame)` treats a frame as captured at the moment of the call, so the timing that reaches the other peer is the timing of your pushing thread. For a live source such as a camera that is exactly right. For a source that has timing of its own, such as a video file, it is not: every hiccup of the executor above ends up in the stream, and audio pushed alongside drifts out of sync with the video. +`pushFrame(VideoFrame)` treats a frame as captured at the moment of the call. The timing that reaches the other peer is therefore the timing of your pushing thread. + +For a live source such as a camera, that is exactly right. For a source with timing of its own, such as a video file, it is not. Every hiccup of the executor above ends up in the stream, and audio pushed alongside drifts out of sync with the video. Pass the capture time instead: From 1e5b0a4acbd0a03a7db6576d5fc303150bba6631 Mon Sep 17 00:00:00 2001 From: Alex Andres Date: Fri, 13 Nov 2026 01:51:44 +0100 Subject: [PATCH 19/37] docs: write the native image page in shorter sentences Same content, long paragraphs split. --- docs/guide/advanced/native-image.md | 8 ++++++-- 1 file changed, 6 insertions(+), 2 deletions(-) diff --git a/docs/guide/advanced/native-image.md b/docs/guide/advanced/native-image.md index 240fd84b..dddec974 100644 --- a/docs/guide/advanced/native-image.md +++ b/docs/guide/advanced/native-image.md @@ -55,7 +55,9 @@ With Gradle, the [Gradle plugin](https://graalvm.github.io/native-build-tools/la The [loading order](/guide/get-started#where-the-native-library-is-loaded-from) is the same as on the JVM, which gives an image two ways to get the library: -**Inside the image.** With the native JAR on the class path at build time, as Maven puts it there, the image carries the library. It writes it to the temporary directory when it starts and loads it from there, as the JVM does from the JAR, so the executable is all there is to ship. Set `webrtc.java.extract.dir` when the temporary directory does not allow running code: +**Inside the image.** With the native JAR on the class path at build time, as Maven puts it there, the image carries the library. The image writes the library to the temporary directory when it starts, and loads it from there, as the JVM does from the JAR. + +The executable is then all there is to ship. Set `webrtc.java.extract.dir` when the temporary directory does not allow running code: ```shell ./app -Dwebrtc.java.extract.dir=/var/lib/app/natives @@ -86,7 +88,9 @@ An image is built for one platform, so it carries the library of the platform it ## Audio Sinks with RTP Packets -An `AudioTrackSink` may implement the `onData` method that also receives [the RTP packets](/guide/audio/headless-audio#who-sent-the-received-audio) the audio was assembled from. Whether a sink does is found out by reflection, which a native image answers only for registered classes. For any other sink, an anonymous class or a lambda among them, the packets are collected: a sink that implements the method gets them, one that does not gets its audio without them, as on the JVM. To spare the collecting, register the sink class for the query: +An `AudioTrackSink` may implement the `onData` method that also receives [the RTP packets](/guide/audio/headless-audio#who-sent-the-received-audio) the audio was assembled from. Whether a sink does is found out by reflection, which a native image answers only for registered classes. + +For any other sink, an anonymous class or a lambda among them, the packets are collected. A sink that implements the method gets them, and one that does not gets its audio without them, as on the JVM. To spare the collecting, register the sink class for the query: ```json [ From 4efc0b588d3f20b9ca78202d355a2ef0cc832da0 Mon Sep 17 00:00:00 2001 From: Alex Andres Date: Fri, 13 Nov 2026 01:52:00 +0100 Subject: [PATCH 20/37] docs: write the peer connection configuration page in shorter sentences Same content, long table cells and paragraphs split. --- docs/guide/networking/peer-connection-config.md | 17 +++++++++++++---- 1 file changed, 13 insertions(+), 4 deletions(-) diff --git a/docs/guide/networking/peer-connection-config.md b/docs/guide/networking/peer-connection-config.md index b9000eea..0af54cab 100644 --- a/docs/guide/networking/peer-connection-config.md +++ b/docs/guide/networking/peer-connection-config.md @@ -22,7 +22,7 @@ Some settings can be changed later with `setConfiguration()`, starting from `get | Setting | Effect | |---|---| | `iceCandidatePoolSize` | Gathers this many candidates before a connection needs them, so that connecting is faster. Default 0. | -| `continualGatheringPolicy` | `GATHER_CONTINUALLY` keeps gathering after the first candidates, so that a network that comes up later gets candidates too, which lets a connection survive a network change. Default `GATHER_ONCE`. | +| `continualGatheringPolicy` | `GATHER_CONTINUALLY` keeps gathering after the first candidates, so that a connection survives a network change (see below). Default `GATHER_ONCE`. | | `tcpCandidatePolicy` | `DISABLED` gathers no TCP candidates. Default `ENABLED`. | | `candidateNetworkPolicy` | `LOW_COST` leaves out cellular networks. Default `ALL`. | | `disableIpv6OnWifi`, `maxIpv6Networks` | Limit IPv6 candidates. Default: IPv6 on Wi-Fi allowed, at most 5 IPv6 networks. | @@ -30,6 +30,8 @@ Some settings can be changed later with `setConfiguration()`, starting from `get | `vpnPreference` | Whether to use, avoid, prefer or require VPN connections. Default `DEFAULT`. | | `surfaceIceCandidatesOnIceTransportTypeChanged` | Signals at once the candidates a change of `iceTransportPolicy` lets through. Default false. | +With `GATHER_CONTINUALLY`, a network that comes up later gets candidates too, which lets a connection survive a network change. + To restrict ports and interfaces, see [Port Allocator Configuration](/guide/networking/port-allocator-config). ## Connectivity Checks @@ -71,21 +73,28 @@ config.cryptoOptions = crypto; `cryptexPolicy` encrypts the RTP header extensions and CSRCs as a whole (RFC 9335) where the peer supports it (`NEGOTIATE`) or always (`REQUIRE`). -`certificates` takes the DTLS certificate a peer connection authenticates with, as an `RTCCertificatePEM`; without one, WebRTC generates a certificate per connection. A certificate that does not parse makes `createPeerConnection()` and `setConfiguration()` throw. The expiration time passed to its constructor is informational: WebRTC takes it from the certificate. The certificates that `getConfiguration()` and `RTCDtlsTransport.getRemoteCertificates()` return carry that time in `getExpires()`, in milliseconds since the Unix epoch, or `RTCCertificatePEM.EXPIRES_UNKNOWN` (-1) if it cannot be read. +`certificates` takes the DTLS certificate a peer connection authenticates with, as an `RTCCertificatePEM`. Without one, WebRTC generates a certificate per connection. A certificate that does not parse makes `createPeerConnection()` and `setConfiguration()` throw. + +The expiration time passed to its constructor is informational: WebRTC takes it from the certificate. The certificates that `getConfiguration()` and `RTCDtlsTransport.getRemoteCertificates()` return carry that time in `getExpires()`, in milliseconds since the Unix epoch. If it cannot be read, `getExpires()` returns `RTCCertificatePEM.EXPIRES_UNKNOWN` (-1). ## Signaling | Setting | Effect | |---|---| | `offerExtmapAllowMixed` | Allows one- and two-byte header extensions to be mixed in offers. Default true. | -| `enableImplicitRollback` | Rolls a pending local offer back when a remote offer arrives, as perfect negotiation needs. Default false. Pair it with `setLocalDescription(observer)`, which creates the offer or answer itself. | +| `enableImplicitRollback` | Rolls a pending local offer back when a remote offer arrives, as perfect negotiation needs. Default false. Pair it with `setLocalDescription(observer)`, which creates the offer or answer. | | `alwaysNegotiateDataChannels` | Includes data channels in offers before any is created. Default false. | ## Changing a Running Connection A few controls are methods of `RTCPeerConnection` rather than configuration fields, and can be used at any time. -`setBitrate(min, current, max)` limits the bandwidth all streams sent by the connection may use together, in bits per second. A `null` value leaves that limit as it is, and `current` resets the bandwidth estimate to that value. Limits from the SDP and from each sender's encoding parameters still apply. Values that are negative or not ordered as min ≤ current ≤ max make it throw. +`setBitrate(min, current, max)` limits the bandwidth that all streams sent by the connection may use together, in bits per second. + +- A `null` value leaves that limit as it is. +- `current` resets the bandwidth estimate to that value. +- Limits from the SDP and from each sender's encoding parameters still apply. +- Values that are negative, or not ordered as min ≤ current ≤ max, make it throw. ```java // Start from 1 Mbps and never use more than 2.5 Mbps. From 382dbeefa8919fddf90bebb9401fc74f2bd0470d Mon Sep 17 00:00:00 2001 From: Alex Andres Date: Fri, 13 Nov 2026 01:52:13 +0100 Subject: [PATCH 21/37] docs: write the camera video page in shorter sentences Same content, long paragraphs split. --- docs/guide/video/camera-capture.md | 17 +++++++++++++---- 1 file changed, 13 insertions(+), 4 deletions(-) diff --git a/docs/guide/video/camera-capture.md b/docs/guide/video/camera-capture.md index 5061c370..a1b8d723 100644 --- a/docs/guide/video/camera-capture.md +++ b/docs/guide/video/camera-capture.md @@ -181,11 +181,15 @@ videoTrack.addSink(new VideoTrackSink() { }); ``` -WebRTC holds a lock while it hands a frame to the sinks, and adding or removing a sink takes the same lock. A change made from within a callback therefore takes effect for the sinks at once, while the native sink is added or removed on another thread right after the callback returned. Disposing the track is the one thing a callback cannot do: `dispose()` throws an `IllegalStateException` there, and belongs on another thread. +WebRTC holds a lock while it hands a frame to the sinks, and adding or removing a sink takes the same lock. A change made from within a callback therefore takes effect for the sinks at once. The native sink is added or removed on another thread, right after the callback returned. + +Disposing the track is the one thing a callback cannot do: `dispose()` throws an `IllegalStateException` there, and belongs on another thread. ### Sharing Frames Between Sinks -By default each sink gets an I420 copy of each frame of its own, which it may write into without the other sinks or the encoder seeing it. That costs a full copy of the pixels per frame and sink: at 1080p and 30 frames a second about 89 MiB a second, for each sink. A sink that only reads the frames can share them instead: +By default, each sink gets an I420 copy of each frame of its own, which it may write into without the other sinks or the encoder seeing it. + +That costs a full copy of the pixels per frame and sink: at 1080p and 30 frames a second, about 89 MiB a second for each sink. A sink that only reads the frames can share them instead: ```java import dev.onvoid.webrtc.media.video.VideoFrameAccess; @@ -193,7 +197,10 @@ import dev.onvoid.webrtc.media.video.VideoFrameAccess; videoTrack.addSink(previewSink, VideoFrameAccess.SHARED); ``` -A shared sink gets the pixels the source handed out, without a copy, which the other shared sinks and the encoder use as well. The planes of its `NativeI420Buffer` are read-only, so a write throws a `ReadOnlyBufferException`; a sink that wants to change a frame takes a writable copy with `frame.copy()`. The frames may come from a pool of the decoder or capturer, so a shared sink should not hold on to them for long: keep a copy instead, and release the frame. +A shared sink gets the pixels the source handed out, without a copy. The other shared sinks and the encoder use the same pixels. + +- The planes of its `NativeI420Buffer` are read-only, so a write throws a `ReadOnlyBufferException`. A sink that wants to change a frame takes a writable copy with `frame.copy()`. +- The frames may come from a pool of the decoder or capturer, so a shared sink should not hold on to them for long. Keep a copy instead, and release the frame. ### Receiving Remote Video Frames @@ -331,7 +338,9 @@ Note that scaling is a computationally expensive operation, especially for high- ### Changing Frames Before They Are Sent -A `VideoProcessor` sits between a camera or desktop source and its tracks, so it can change what is sent and shown: blur a background, draw an overlay, mirror the picture. The source passes each frame to `onFrameCaptured()` after adapting it to what its tracks ask for, so frames already have the size and rate that will be sent. The processor delivers the frames to send to the sink it gets in `setSink()`: the same frame, a new one, or none to drop it. +A `VideoProcessor` sits between a camera or desktop source and its tracks, so it can change what is sent and shown: blur a background, draw an overlay, mirror the picture. + +The source passes each frame to `onFrameCaptured()` after adapting it to what its tracks ask for, so frames already have the size and rate that will be sent. The processor delivers the frames to send to the sink it gets in `setSink()`: the same frame, a new one, or none to drop it. ```java videoSource.setVideoProcessor(new VideoProcessor() { From e457f3e885cfd5e59a6a932d30dcd717e474191a Mon Sep 17 00:00:00 2001 From: Alex Andres Date: Fri, 13 Nov 2026 01:52:21 +0100 Subject: [PATCH 22/37] docs: write the custom audio source page in shorter sentences Same content, long sentences split. --- docs/guide/audio/custom-audio-source.md | 8 +++++--- 1 file changed, 5 insertions(+), 3 deletions(-) diff --git a/docs/guide/audio/custom-audio-source.md b/docs/guide/audio/custom-audio-source.md index 74b43dd2..d97cd19b 100644 --- a/docs/guide/audio/custom-audio-source.md +++ b/docs/guide/audio/custom-audio-source.md @@ -48,7 +48,7 @@ AudioTrack audioTrack = factory.createAudioTrack("audio-track-id", audioSource); ``` ::: tip Pushed and device-captured audio side by side -Each audio sender gets the audio of its own track, so one factory, and one peer connection, can send a `CustomAudioSource` track beside a microphone track created with `createAudioSource(AudioOptions)`, and beside a track forwarded from a remote peer. +Each audio sender gets the audio of its own track. One factory, and one peer connection, can therefore send a `CustomAudioSource` track beside a microphone track created with `createAudioSource(AudioOptions)`, and beside a track forwarded from a remote peer. The recording device is opened only while a microphone track is sent. A factory that sends only pushed audio never opens it. Playout of received audio is unaffected. ::: @@ -76,7 +76,7 @@ byte[] audioData = new byte[frameCount * channels * bytesPerSample]; audioSource.pushAudio(audioData, bitsPerSample, sampleRate, channels, frameCount); ``` -The call checks its arguments and throws `IllegalArgumentException` rather than letting a bad chunk reach WebRTC, where it would read past your array, abort the process, or be dropped by the encoder while the push appears to succeed: +The call checks its arguments, and throws `IllegalArgumentException` rather than let a bad chunk reach WebRTC. There, the chunk would read past your array, abort the process, or be dropped by the encoder while the push appears to succeed: - Samples must be **16-bit** signed PCM in the platform byte order, with the channels interleaved. - The array must hold at least `frameCount * channels * 2` bytes. A longer array is fine; only that many bytes are read. @@ -101,7 +101,9 @@ audioSource.pushAudio(pcm, 16, sampleRate, channels, frameCount, baseUs + presentationUs); ``` -Use the same base for the audio and the video of one source, and the two stay in sync on the receiving side. See [Custom Video Source](/guide/video/custom-video-source) for the video half. Chunks must still be pushed in real time; the timestamp describes the audio, it does not schedule it. A chunk pushed without a timestamp is stamped with the same clock, so a source may mix the two calls. +Use the same base for the audio and the video of one source, and the two stay in sync on the receiving side. See [Custom Video Source](/guide/video/custom-video-source) for the video half. + +Chunks must still be pushed in real time: the timestamp describes the audio, it does not schedule it. A chunk pushed without a timestamp is stamped with the same clock, so a source may mix the two calls. ## Audio Format Considerations From f2cb3ca234180c06bc3decd124bc8edf88349b88 Mon Sep 17 00:00:00 2001 From: Alex Andres Date: Fri, 13 Nov 2026 01:52:32 +0100 Subject: [PATCH 23/37] docs: write the headless audio page in shorter sentences Same content, the effects of the old capture path as a list. --- docs/guide/audio/headless-audio.md | 8 ++++++-- 1 file changed, 6 insertions(+), 2 deletions(-) diff --git a/docs/guide/audio/headless-audio.md b/docs/guide/audio/headless-audio.md index 88cae83d..14bac3f3 100644 --- a/docs/guide/audio/headless-audio.md +++ b/docs/guide/audio/headless-audio.md @@ -85,8 +85,12 @@ A receiver learns its sources as playout pulls the audio too, so `RTCRtpReceiver The module has no capture path. There is no device to capture from, so `initRecording()` and `startRecording()` only move the module through the recording lifecycle and deliver no audio. Audio you want to send goes through a `CustomAudioSource`, which hands it to the track's senders directly. ::: info -Earlier versions pulled the render mix back through the module and handed it to WebRTC as captured audio. That made a peer connection send the audio it had just received from the remote peer straight back to it, and it fed the send stream a second time alongside the pushed audio, which aborted the process inside WebRTC. Calling `startRecording()` is now harmless but does nothing. -::: +Earlier versions pulled the render mix back through the module, and handed it to WebRTC as captured audio. That had two effects: + +- A peer connection sent the audio it had just received from the remote peer straight back to it. +- The send stream got the audio a second time, alongside the pushed audio, which aborted the process inside WebRTC. + +Calling `startRecording()` is now harmless, but does nothing. Typical steps: From c11f994c5bda8982d3761c0a89a6f45123cdac31 Mon Sep 17 00:00:00 2001 From: Alex Andres Date: Fri, 13 Nov 2026 01:52:39 +0100 Subject: [PATCH 24/37] docs: write the file transfer page in shorter sentences Same content, long paragraphs split. --- docs/guide/data/file-transfer.md | 13 ++++++++++--- 1 file changed, 10 insertions(+), 3 deletions(-) diff --git a/docs/guide/data/file-transfer.md b/docs/guide/data/file-transfer.md index 18af21f3..a1543b0d 100644 --- a/docs/guide/data/file-transfer.md +++ b/docs/guide/data/file-transfer.md @@ -18,7 +18,10 @@ peer.onChannel(channel -> { }); ``` -`FileTransfers` takes over the channel: nothing else may be sent on it. Files go one at a time in each direction, in the order they were sent, and both peers may send at the same time. Each side reads and writes its files and runs its handlers on a thread of its own, not on the peer's event thread. Those handlers must not block. +`FileTransfers` takes over the channel: nothing else may be sent on it. + +- Files go one at a time in each direction, in the order they were sent, and both peers may send at the same time. +- Each side reads and writes its files, and runs its handlers, on a thread of its own, not on the peer's event thread. Those handlers must not block. ## Confirmation, not just handing data to the transport @@ -30,9 +33,13 @@ The future that `DataChannel.send()` returns completes once the data has been ha ## Offers -An offer's `name()` and `size()` come from the remote peer. Don't turn the name into a path as it is, because it may contain `../` or path separators. `safeFileName()` gives you a single file name that stays in the directory you resolve it against. `reject(reason)` makes the sender's transfer fail with that reason, and the sender's next file follows. If an offer arrives before any handler is registered, it waits for the first handler. +An offer's `name()` and `size()` come from the remote peer. Don't turn the name into a path as it is, because it may contain `../` or path separators. `safeFileName()` gives you a single file name that stays in the directory you resolve it against. + +`reject(reason)` makes the sender's transfer fail with that reason, and the sender's next file follows. If an offer arrives before any handler is registered, it waits for the first handler. + +The data goes to `partialFile()` next to the target, and that file replaces the target only after the checksum matched. -The data goes to `partialFile()` next to the target, and that file replaces the target only after the checksum matched. If more data arrives than the offer announced, or more than the window allows, or the data is out of order, the receiver treats it as a protocol error: the transfer fails and the channel closes. +The receiver treats it as a protocol error if more data arrives than the offer announced, more than the window allows, or the data arrives out of order. The transfer then fails, and the channel closes. ## Cancelling and failures From 92c5a2ccf4c04b5091770418bbe9b139ca4d25fd Mon Sep 17 00:00:00 2001 From: Alex Andres Date: Fri, 13 Nov 2026 01:52:57 +0100 Subject: [PATCH 25/37] docs: write the release notes of the build page in shorter sentences Same content, long paragraphs split. --- docs/guide/build.md | 19 ++++++------------- 1 file changed, 6 insertions(+), 13 deletions(-) diff --git a/docs/guide/build.md b/docs/guide/build.md index 67cd5d98..2f481769 100644 --- a/docs/guide/build.md +++ b/docs/guide/build.md @@ -116,13 +116,9 @@ caches, so a native rebuild also includes the updated patches. ## Release workflow -Maintainers start **Prepare release** (`release.yml`) from `main`, providing the -next development version, such as `0.16.0-SNAPSHOT`, and optionally the version -to release, such as `0.15.0`. Without one, the release takes the root pom's -version without `-SNAPSHOT`. Preparation commits the changelog and version -changes and atomically pushes `main` and the release tag. It then dispatches **Publish release** (`release-publish.yml`) on that tag using -`GITHUB_TOKEN` with `actions: write`; no additional token is needed. A push made -with `GITHUB_TOKEN` does not trigger a tag push workflow on its own. +Maintainers start **Prepare release** (`release.yml`) from `main`. They provide the next development version, such as `0.16.0-SNAPSHOT`, and optionally the version to release, such as `0.15.0`. Without one, the release takes the root pom's version without `-SNAPSHOT`. + +Preparation commits the changelog and version changes, and atomically pushes `main` and the release tag. It then dispatches **Publish release** (`release-publish.yml`) on that tag, using `GITHUB_TOKEN` with `actions: write`; no additional token is needed. A push made with `GITHUB_TOKEN` does not trigger a tag push workflow on its own. Publication requires a `vX.Y.Z` tag whose commit belongs to `main` and whose version matches the root pom. All platform builds, native tests and publication @@ -135,12 +131,9 @@ it as `deployment-bundle-dry-run`. It pushes no commits or tags, dispatches no publication workflow, and uses placeholder Central credentials with uploads disabled. -The jars are reproducible: every entry and the manifest's `Build-Date` carry -`project.build.outputTimestamp` from the root pom instead of the time of the -build, so the same commit builds into the same Java jars. Release preparation -sets it to the time of the release commit and again for the next development -version; it needs no manual change. The native library jars are not covered, -since CMake and the compilers do not build byte for byte the same libraries. +The jars are reproducible. Every entry and the manifest's `Build-Date` carry `project.build.outputTimestamp` from the root pom instead of the time of the build, so the same commit builds into the same Java jars. + +Release preparation sets that timestamp to the time of the release commit, and again for the next development version; it needs no manual change. The native library jars are not covered, since CMake and the compilers do not build byte for byte the same libraries. If preparation pushed the tag but dispatch failed, start Publish release manually and select that tag. Before retrying a failed publication, check whether Maven From 9b1e1d75160cfbd2d8a9beabedc4c772b517d005 Mon Sep 17 00:00:00 2001 From: Alex Andres Date: Fri, 13 Nov 2026 01:52:57 +0100 Subject: [PATCH 26/37] docs: split a long paragraph of the examples page Same content. --- docs/guide/examples.md | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/docs/guide/examples.md b/docs/guide/examples.md index 82452ea7..2adfb1d6 100644 --- a/docs/guide/examples.md +++ b/docs/guide/examples.md @@ -173,7 +173,9 @@ mvn exec:java -D"exec.mainClass=dev.onvoid.webrtc.examples.MediaFileExample" -D" ## Media File Player -The [`MediaFilePlayerExample`](https://github.com/devopvoid/webrtc-java/blob/main/webrtc-examples/src/main/java/dev/onvoid/webrtc/examples/MediaFilePlayerExample.java) is a Swing application built on the [Client API](/guide/client-api). It sends a media file, or a live RTSP stream, from one peer to another in the same process, then shows the received video and plays the received audio in sync, and records what is received. Start and Stop buttons set up and tear down the whole session, and Record starts and stops a recording. +The [`MediaFilePlayerExample`](https://github.com/devopvoid/webrtc-java/blob/main/webrtc-examples/src/main/java/dev/onvoid/webrtc/examples/MediaFilePlayerExample.java) is a Swing application built on the [Client API](/guide/client-api). It sends a media file, or a live RTSP stream, from one peer to another in the same process. It then shows the received video, plays the received audio in sync, and records what is received. + +Start and Stop buttons set up and tear down the whole session, and Record starts and stops a recording. **Key features demonstrated:** - Sending the sources of a `MediaFileSource` as client API tracks, with `videoTrack` and `audioTrack` From 623c8f36db54723d5711d6c4a6407989b89b91e1 Mon Sep 17 00:00:00 2001 From: Alex Andres Date: Fri, 13 Nov 2026 01:52:57 +0100 Subject: [PATCH 27/37] docs: write the speech detection rules of audio processing as a list Same content. --- docs/guide/audio/audio-processing.md | 9 ++++++++- 1 file changed, 8 insertions(+), 1 deletion(-) diff --git a/docs/guide/audio/audio-processing.md b/docs/guide/audio/audio-processing.md index 4a266bd9..bf4bbdf7 100644 --- a/docs/guide/audio/audio-processing.md +++ b/docs/guide/audio/audio-processing.md @@ -315,7 +315,14 @@ factory.addVoiceActivityListener(new VoiceActivityConfig(0.5, Duration.ofMillis( factory.removeVoiceActivityListener(listener); ``` -It works with the `AudioProcessing` the factory was built with as well as with the one it builds itself. All listeners share one detector, and each decides with the threshold and hangover of its own `VoiceActivityConfig`. Audio arrives while the audio device module records, which WebRTC starts once a peer connection sends a microphone track. A listener must not add or remove a listener. Disposing of the factory removes the listeners it still has. The [Client API](/guide/client/receiving-media#audio) offers the same through `LocalAudio.onVoiceActivity`. +It works with the `AudioProcessing` the factory was built with, as well as with the one it builds itself. + +- All listeners share one detector, and each decides with the threshold and hangover of its own `VoiceActivityConfig`. +- Audio arrives while the audio device module records, which WebRTC starts once a peer connection sends a microphone track. +- A listener must not add or remove a listener. +- Disposing of the factory removes the listeners it still has. + +The [Client API](/guide/client/receiving-media#audio) offers the same through `LocalAudio.onVoiceActivity`. ## Statistics From 033e44cb696990c4f13ba333843c49a38fbd33f2 Mon Sep 17 00:00:00 2001 From: Alex Andres Date: Fri, 13 Nov 2026 01:53:13 +0100 Subject: [PATCH 28/37] docs: write the client data channels page in shorter sentences Same content, the receive limit and message previews as lists. --- docs/guide/client/data-channels.md | 21 ++++++++++++++------- 1 file changed, 14 insertions(+), 7 deletions(-) diff --git a/docs/guide/client/data-channels.md b/docs/guide/client/data-channels.md index fb51a7e8..46fc5a2e 100644 --- a/docs/guide/client/data-channels.md +++ b/docs/guide/client/data-channels.md @@ -34,7 +34,15 @@ for (byte[] chunk : chunks) { A sender that does not wait still cannot fill Java memory: each channel holds at most 16 MiB waiting to be sent, and a `send` beyond that fails at once with a `ChannelFullException`. `PeerOptions.builder().channelSendLimit(bytes)` changes the limit for every channel of a peer. `bufferedAmount()` tells how much a channel holds. -Received messages are bounded as well. WebRTC cannot be told to stop receiving, so messages that arrive faster than the handlers take them, or while a channel has no handler, wait in Java. Each channel lets at most 16 MiB of them wait. When more arrive, the channel closes instead of dropping messages unnoticed: the peer's `onError` handlers hear why, the messages after the one over the limit are not delivered, and both sides see the channel close. `PeerOptions.builder().channelReceiveLimit(bytes)` changes the limit. A handler that takes long should hand the work to a thread of its own. +Received messages are bounded as well. WebRTC cannot be told to stop receiving, so messages wait in Java when they arrive faster than the handlers take them, or while a channel has no handler. Each channel lets at most 16 MiB of them wait. + +When more arrive, the channel closes, instead of dropping messages unnoticed: + +- The peer's `onError` handlers hear why. +- The messages after the one over the limit are not delivered. +- Both sides see the channel close. + +`PeerOptions.builder().channelReceiveLimit(bytes)` changes the limit. A handler that takes long should hand the work to a thread of its own. ## Channel Options @@ -59,12 +67,11 @@ A completed `send` future means the data was handed to the transport, not that t ### Message Previews -`Message.toString()` reports only the message type and byte length. Access content -explicitly with `text()` or `bytes()`. For diagnostics, `message.preview(64)` inspects -at most 64 payload bytes and escapes control characters. Binary previews use hex. -Use `message.preview(64, prefix -> redact(prefix))` to apply your own masking before -escaping; the masking function must handle secrets truncated at the prefix boundary. -The preview marks truncated content and also limits the masking function's output. +`Message.toString()` reports only the message type and byte length. Access the content explicitly, with `text()` or `bytes()`. + +- For diagnostics, `message.preview(64)` inspects at most 64 payload bytes, and escapes control characters. Binary previews use hex. +- `message.preview(64, prefix -> redact(prefix))` applies your own masking before escaping. The masking function must handle secrets truncated at the prefix boundary. +- The preview marks truncated content, and also limits the masking function's output. ### Option Ranges From 3b771a855d8f10e88648ee1308f4ba51985ae30b Mon Sep 17 00:00:00 2001 From: Alex Andres Date: Fri, 13 Nov 2026 01:53:13 +0100 Subject: [PATCH 29/37] docs: split a long paragraph of the core data channels page Same content. --- docs/guide/data/data-channels.md | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/docs/guide/data/data-channels.md b/docs/guide/data/data-channels.md index 7b5e966b..c4b0793f 100644 --- a/docs/guide/data/data-channels.md +++ b/docs/guide/data/data-channels.md @@ -205,7 +205,9 @@ dataChannel.sendAsync(binaryChannelBuffer, new RTCDataChannelSendObserver() { }); ``` -The observer is called exactly once, with a failure if the operation is discarded while the channel shuts down. Success means the message was queued locally, not that the peer received it. Callbacks run on a native callback thread shared with other WebRTC observers; they may call other WebRTC methods, but should not block, so dispatch long work to an executor of your own. Without an observer, `sendAsync` logs failures and reports nothing to the caller. +The observer is called exactly once, with a failure if the operation is discarded while the channel shuts down. Success means the message was queued locally, not that the peer received it. + +Callbacks run on a native callback thread shared with other WebRTC observers. They may call other WebRTC methods, but should not block, so dispatch long work to an executor of your own. Without an observer, `sendAsync` logs failures and reports nothing to the caller. ### Receiving Data From acdb8a967d317f9b48563cf47857cab330e0c335 Mon Sep 17 00:00:00 2001 From: Alex Andres Date: Fri, 13 Nov 2026 01:53:32 +0100 Subject: [PATCH 30/37] docs: split the introduction of the media devices page Same content. --- docs/guide/media/media-devices.md | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/docs/guide/media/media-devices.md b/docs/guide/media/media-devices.md index 8e58b613..a7752531 100644 --- a/docs/guide/media/media-devices.md +++ b/docs/guide/media/media-devices.md @@ -8,7 +8,9 @@ This guide explains how to work with media devices (microphones, speakers, and c - Getting device capabilities - Listening for device hotplug events -Cameras and microphones play a key role in WebRTC. In a more complex application, you will most likely want to check all the connected cameras and microphones and select the appropriate device for the WebRTC session. These devices can be enumerated with the class `MediaDevices`. With `MediaDevices` you can also listen for device changes, whenever a device is connected or disconnected. +Cameras and microphones play a key role in WebRTC. A more complex application will most likely check all connected cameras and microphones, and select the appropriate device for the WebRTC session. + +The class `MediaDevices` enumerates these devices. It can also tell you about device changes, whenever a device is connected or disconnected. ## Querying Media Devices From 58a6c4e7671cc14a1cc1b0189e8c7608fd7e208d Mon Sep 17 00:00:00 2001 From: Alex Andres Date: Fri, 13 Nov 2026 01:53:32 +0100 Subject: [PATCH 31/37] docs: split a long sentence of the field trials page Same content. --- docs/guide/advanced/field-trials.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/guide/advanced/field-trials.md b/docs/guide/advanced/field-trials.md index b1e9209b..70eb2045 100644 --- a/docs/guide/advanced/field-trials.md +++ b/docs/guide/advanced/field-trials.md @@ -41,7 +41,7 @@ Each key and value must be non-null and non-empty. Passing a map that violates t ## Tips and Troubleshooting -- Field trial names and accepted groups (e.g. `Enabled`, `Disabled`) are defined by the underlying WebRTC native code and change between WebRTC releases; consult the WebRTC source for the branch this library is built against to find currently available trials. +- Field trial names and accepted groups (e.g. `Enabled`, `Disabled`) are defined by the underlying WebRTC native code, and change between WebRTC releases. To find the trials available now, consult the WebRTC source of the branch this library is built against. - Since field trials configure experimental or internal behavior, prefer enabling only the trials you specifically need, and re-verify them after upgrading to a new release of this library. ## Related API From 0fa618fcf08672b47165e58522776b7091211e73 Mon Sep 17 00:00:00 2001 From: Alex Andres Date: Fri, 13 Nov 2026 01:53:33 +0100 Subject: [PATCH 32/37] docs: split the macOS paragraph of the desktop video page Same content. --- docs/guide/video/desktop-capture.md | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/docs/guide/video/desktop-capture.md b/docs/guide/video/desktop-capture.md index 09cf1421..92317c17 100644 --- a/docs/guide/video/desktop-capture.md +++ b/docs/guide/video/desktop-capture.md @@ -188,7 +188,9 @@ windowCapturer.dispose(); ### Capturing on macOS -On macOS 14 and newer, screens and windows are captured with ScreenCaptureKit; older releases use the earlier Core Graphics capture. Both need the app to have Screen Recording permission (System Settings → Privacy & Security → Screen & System Audio Recording). Without it, the windows of other apps are listed without their titles and their content cannot be captured. Windows of the app's own process need no permission, and are captured with Core Graphics. +On macOS 14 and newer, screens and windows are captured with ScreenCaptureKit; older releases use the earlier Core Graphics capture. Both need the app to have Screen Recording permission (System Settings → Privacy & Security → Screen & System Audio Recording). + +Without that permission, the windows of other apps are listed without their titles, and their content cannot be captured. Windows of the app's own process need no permission, and are captured with Core Graphics. While a captured window is minimized or hidden, its last image repeats. When it is closed, capture stops. From d4669cdb8531fc5505b56b8d1dde10cbd7894181 Mon Sep 17 00:00:00 2001 From: Alex Andres Date: Fri, 13 Nov 2026 01:53:47 +0100 Subject: [PATCH 33/37] docs: write the adapter notes of voice activity detection as a list Same content. --- docs/tools/audio/voice-activity-detector.md | 12 +++++------- 1 file changed, 5 insertions(+), 7 deletions(-) diff --git a/docs/tools/audio/voice-activity-detector.md b/docs/tools/audio/voice-activity-detector.md index f1fdd913..3a3b04f7 100644 --- a/docs/tools/audio/voice-activity-detector.md +++ b/docs/tools/audio/voice-activity-detector.md @@ -220,13 +220,11 @@ sinks. Its speech comes from a listener on the factory, see [`AudioRecorder`](/tools/audio/audio-recorder) captures a microphone on its own, apart from any peer connection. -The adapter invokes the consumer on the thread that delivers the audio: a thread -of the `onAudio` subscription, or the audio callback thread of a sink. Keep work -there short. A subscription that falls behind loses the oldest audio beyond its -buffer, so a detector that must stay live passes a short one, as in -`onAudio(Duration.ofMillis(200), vad)`. Apply application-level timing to produce stable speech-start/speech-end -events, rather than toggling a UI or recording for every frame. Audio format -changes clear the adapter's partial frame and reset its detector. +The adapter invokes the consumer on the thread that delivers the audio: a thread of the `onAudio` subscription, or the audio callback thread of a sink. Keep work there short. + +- A subscription that falls behind loses the oldest audio beyond its buffer. A detector that must stay live therefore passes a short one, as in `onAudio(Duration.ofMillis(200), vad)`. +- Apply application-level timing to produce stable speech-start and speech-end events, rather than toggling a UI or recording for every frame. +- Audio format changes clear the adapter's partial frame, and reset its detector. ## Related guides From b41646ba1db6a789f1c16408f1c0a1835c9fdadc Mon Sep 17 00:00:00 2001 From: Alex Andres Date: Fri, 13 Nov 2026 01:53:47 +0100 Subject: [PATCH 34/37] docs: split a long sentence of the video buffer converter page Same content. --- docs/tools/video/video-buffer-converter.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/tools/video/video-buffer-converter.md b/docs/tools/video/video-buffer-converter.md index 1e3b243d..84422546 100644 --- a/docs/tools/video/video-buffer-converter.md +++ b/docs/tools/video/video-buffer-converter.md @@ -37,7 +37,7 @@ Consult the FourCC enum in your version for the complete list. ## Buffer sizing -Frames are stored without row padding, except where a format groups pixels in pairs: a YUY2 or UYVY row, and an NV12 or NV21 chroma row, always ends with a whole pair, so an odd width is rounded up to the next pair. A buffer must hold at least: +Frames are stored without row padding, except where a format groups pixels in pairs. A YUY2 or UYVY row, and an NV12 or NV21 chroma row, always ends with a whole pair, so an odd width is rounded up to the next pair. A buffer must hold at least: - `ARGB`, `BGRA`, `ABGR`, `RGBA`: `width * height * 4` bytes - `BG24`: `width * height * 3` bytes - `I420`, `NV12`, `NV21`: `width * height + 2 * ((width + 1) / 2) * ((height + 1) / 2)` bytes From 618038761219e0026bf161e8acbc38d9ced2b7b1 Mon Sep 17 00:00:00 2001 From: Alex Andres Date: Fri, 13 Nov 2026 01:53:47 +0100 Subject: [PATCH 35/37] docs: split a paragraph of the sending media page Same content. --- docs/guide/client/sending-media.md | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/docs/guide/client/sending-media.md b/docs/guide/client/sending-media.md index 431338f3..27f022d5 100644 --- a/docs/guide/client/sending-media.md +++ b/docs/guide/client/sending-media.md @@ -90,7 +90,9 @@ CustomAudio voice = media.customAudio("voice"); voice.push(pcm, PcmFormat.of(48000, 1)); ``` -`CustomAudio` takes PCM in pieces of any length and from any thread. It cuts the audio into the 10 ms chunks WebRTC works with, and sends one every 10 ms from a thread of its own. Pushing a whole second at once is therefore as fine as pushing every 10 ms. `queuedMillis()` tells how much is still waiting, and `clear()` drops it. +`CustomAudio` takes PCM in pieces of any length and from any thread. It cuts the audio into the 10 ms chunks WebRTC works with, and sends one every 10 ms from a thread of its own. Pushing a whole second at once is therefore as fine as pushing every 10 ms. + +`queuedMillis()` tells how much is still waiting, and `clear()` drops it. A producer faster than real time would otherwise fill memory, and push the delay up without end. The queue therefore holds at most 10 seconds. A `push` beyond that fails with an `AudioQueueFullException`, and none of its audio is kept. `queueLimit(max, overflow)` sets another limit, and what happens beyond it: From 8db7cd5fe7da3f37af57b04bcb2c9cbf900de16f Mon Sep 17 00:00:00 2001 From: Alex Andres Date: Fri, 13 Nov 2026 01:54:38 +0100 Subject: [PATCH 36/37] docs: correct device query failures on the client API design page A failed device query, also for an unavailable audio backend, throws MediaException; it does not come back empty. Also names screens() and windows() as asking the system on every call, and lists the blocking calls below the thread table. --- docs/guide/client-api-design.md | 16 ++++++++++++---- 1 file changed, 12 insertions(+), 4 deletions(-) diff --git a/docs/guide/client-api-design.md b/docs/guide/client-api-design.md index 75bb5557..b6e0b58f 100644 --- a/docs/guide/client-api-design.md +++ b/docs/guide/client-api-design.md @@ -140,7 +140,7 @@ and it hands on to the event executor what the application should hear. | Code | Runs on | May it block? | |---|---|---| -| Calls into the API | the caller's thread, any thread | They don't block, with these exceptions. `createPeer` waits for the connection, so that an invalid configuration fails there. The `LocalMedia` methods that make a track and `LocalTrack.setEnabled` and `LocalVideo.setContentHint` wait briefly for a WebRTC thread, since the factory and the tracks are proxies. `camera()` and `screen()` also open their capture on the caller's thread. The first `devices()`, `screens()` and `windows()` ask the system there. `WebRtc.close()` waits up to `closeTimeout`, and a `join()` the caller writes waits as written. | +| Calls into the API | the caller's thread, any thread | No, apart from the exceptions below the table. | | Event handlers, future callbacks, `Signaling.send` | the peer's event thread, or the configured `eventExecutor` | Yes, but it only delays that peer's events. It must not `join()` a future of this API: a direct `join()`/`get()` throws `BlockingCallException`. | | `LocalTrack.onEnded`, `LocalAudio.onVoiceActivity`, `Devices.onChange`, futures of local tracks and devices | the event thread of the runtime's local media, or the configured `eventExecutor` | Yes, but it delays the other events of local media. The rule against `join()` holds here too. | | Frame handlers (`onFrame`, `onAudio`) | a thread of each subscription, from the frame pool | Better not. A slow video handler gets the newest frame next, and the frames in between are dropped and released. A slow audio handler loses the oldest audio beyond its buffer. | @@ -148,6 +148,14 @@ and it hands on to the event executor what the application should hear. | Native calls | internal control threads | Not application code. Each operation has a timeout (default 30 s), so a stuck native call fails with a `TimeoutException` instead of hanging. The peer then brings signaling back to stable, and gives up if that fails too. | | WHIP and WHEP requests | `java.net.http.HttpClient`, never the control pool | Not application code. A slow server holds up no peer. | | `FileTransfers` handlers, file reads and writes | a thread of each `FileTransfers`, never the peer's event thread | Yes, but it delays the other transfers of that channel. | + +Calls into the API don't block, with these exceptions: + +- `createPeer` waits for the connection, so that an invalid configuration fails there. +- The `LocalMedia` methods that make a track, `LocalTrack.setEnabled` and `LocalVideo.setContentHint` wait briefly for a WebRTC thread, since the factory and the tracks are proxies. +- `camera()` and `screen()` also open their capture on the caller's thread. +- The first `devices()`, and every `screens()` and `windows()`, ask the system there. +- `WebRtc.close()` waits up to `closeTimeout`, and a `join()` the caller writes waits as written. | WebRTC callbacks | signaling, network and capture threads | Not application code. The library copies and enqueues only. | ### Guarantees @@ -614,9 +622,9 @@ ScreenOptions screen = ScreenOptions.screen(rtc.media().devices().screens().get( current one if the new one fails too. - The device lists are snapshots, read once by the first `devices()` call, on the caller's thread. Where the system reports no device changes, as a Linux machine without PulseAudio, they are read on - each call instead. `screens()` and `windows()` always ask the system. A kind of device the system - cannot list comes back empty, and any other failed query throws `MediaException`, so that an empty - list means there are no such devices. + each call instead. `screens()` and `windows()` always ask the system. A query that fails, also + for an unavailable audio backend, throws `MediaException`, so that an empty list means there are + no such devices. - `onVoiceActivity` analyzes the audio in native code, and only the changes reach Java. ### `Sender`, `RemoteTrack` From 36bb577278a56ca50b6f2fc8216118309da7695b Mon Sep 17 00:00:00 2001 From: Alex Andres Date: Fri, 13 Nov 2026 01:56:56 +0100 Subject: [PATCH 37/37] docs: drop the conclusion sections They restated the page. The advice two of them held stays as Best Practices, and the CPU note of audio processing joins its Best Practices. --- docs/guide/audio/audio-devices.md | 7 ------- docs/guide/audio/audio-processing.md | 6 +----- docs/guide/audio/custom-audio-source.md | 6 ------ docs/guide/audio/dtmf-sender.md | 8 -------- docs/guide/data/data-channels.md | 9 --------- docs/guide/media/constraints.md | 13 ------------- docs/guide/media/media-devices.md | 7 +------ docs/guide/monitoring/logging.md | 3 --- docs/guide/monitoring/rtc-stats.md | 4 ---- docs/guide/video/camera-capture.md | 6 ------ docs/guide/video/custom-video-source.md | 6 ------ docs/guide/video/desktop-capture.md | 7 +------ 12 files changed, 3 insertions(+), 79 deletions(-) diff --git a/docs/guide/audio/audio-devices.md b/docs/guide/audio/audio-devices.md index 86207585..8258f5c7 100644 --- a/docs/guide/audio/audio-devices.md +++ b/docs/guide/audio/audio-devices.md @@ -71,7 +71,6 @@ Pass the configured AudioDeviceModule to the PeerConnectionFactory constructor: PeerConnectionFactory factory = new PeerConnectionFactory(audioModule); ``` - ## Disabling Audio If you want to completely disable audio processing in your WebRTC application, you can create an AudioDeviceModule with the `kDummyAudio` layer: @@ -142,9 +141,3 @@ MediaDevices.addDeviceChangeListener(event -> { // Update UI or device selection as needed }); ``` - -## Conclusion - -This guide has walked you through the process of implementing audio device selection in your WebRTC application. -By implementing these capabilities, your application can provide users with greater control over their audio experience, accommodating different hardware setups and preferences. -For more advanced audio processing options, consider exploring the audio processing APIs available in this documentation. \ No newline at end of file diff --git a/docs/guide/audio/audio-processing.md b/docs/guide/audio/audio-processing.md index bf4bbdf7..68703dcf 100644 --- a/docs/guide/audio/audio-processing.md +++ b/docs/guide/audio/audio-processing.md @@ -361,8 +361,4 @@ These statistics are particularly useful for monitoring the performance of echo 5. **Monitor statistics**: Use the statistics to monitor the performance of echo cancellation and adjust settings if needed. -## Conclusion - -The `AudioProcessing` class provides powerful audio processing capabilities for real-time communications. By properly configuring and using these features, you can significantly improve the audio quality in your applications. - -Remember that audio processing is CPU-intensive, so consider the performance implications when enabling multiple features, especially on resource-constrained devices. +6. **Mind the CPU**: Audio processing is CPU-intensive, so consider the performance implications when enabling multiple features, especially on resource-constrained devices. diff --git a/docs/guide/audio/custom-audio-source.md b/docs/guide/audio/custom-audio-source.md index d97cd19b..32101556 100644 --- a/docs/guide/audio/custom-audio-source.md +++ b/docs/guide/audio/custom-audio-source.md @@ -300,9 +300,3 @@ The canceller can only remove echo of audio that actually went through `processR ::: tip If a factory sends only pushed audio, the processing module WebRTC creates for that factory does no work. Enabling its features through `AudioOptions` or the factory's `AudioProcessing` changes nothing for a `CustomAudioSource`; only the instance you apply yourself does. ::: - -## Conclusion - -The `CustomAudioSource` provides a flexible way to integrate external audio sources with WebRTC. By understanding the audio format parameters and properly managing the audio data flow, you can create applications that use custom audio from virtually any source. - -For more advanced audio processing options, consider exploring the audio processing APIs available in this documentation. \ No newline at end of file diff --git a/docs/guide/audio/dtmf-sender.md b/docs/guide/audio/dtmf-sender.md index bc80c3f0..045287d3 100644 --- a/docs/guide/audio/dtmf-sender.md +++ b/docs/guide/audio/dtmf-sender.md @@ -167,11 +167,3 @@ sender.dispose(); - For IVR systems that might need more processing time, consider longer durations 5. **Tone Buffering**: Be aware that tones are queued and played sequentially. If you need to cancel queued tones, you can call `insertDtmf("")` to clear the queue. - -## Conclusion - -The RTCDtmfSender provides a standard way to send DTMF tones over WebRTC audio connections. This functionality is particularly useful for applications that need to interact with traditional telephony systems, IVR systems, or any service that uses DTMF for signaling. - -By following the guidelines in this document, you can effectively integrate DTMF functionality into your WebRTC applications, enabling users to interact with automated systems or trigger actions using their device's keypad. - -For more information on other WebRTC features, refer to the additional guides in the documentation. \ No newline at end of file diff --git a/docs/guide/data/data-channels.md b/docs/guide/data/data-channels.md index c4b0793f..26598fca 100644 --- a/docs/guide/data/data-channels.md +++ b/docs/guide/data/data-channels.md @@ -324,12 +324,3 @@ Unlike other disposable objects, a data channel is not disposed by `close()`, wh 7. **Binary vs. Text**: Use the appropriate flag when creating `RTCDataChannelBuffer` objects: - `false` for UTF-8 text data - `true` for binary data - -## Conclusion - -WebRTC data channels provide a powerful way to establish peer-to-peer communication for transferring arbitrary data between clients. -Data channels complement WebRTC's audio and video capabilities, making it possible to build comprehensive real-time applications that include text chat, file transfers, game state synchronization, and other custom data exchange requirements. - -For optimal performance, remember to follow the best practices outlined in this guide, particularly regarding buffer management and proper cleanup of resources. - -For more information on other WebRTC features, refer to the additional guides in the documentation. diff --git a/docs/guide/media/constraints.md b/docs/guide/media/constraints.md index 4c00013f..25c2def4 100644 --- a/docs/guide/media/constraints.md +++ b/docs/guide/media/constraints.md @@ -218,16 +218,3 @@ sender.setParameters(parameters); ``` For audio, `adaptivePtime` lets the encoder send longer packets when the bitrate is low, which saves the overhead of many small packets. - -## Conclusion - -In this guide, we've explored several important techniques for controlling media quality and bandwidth usage in WebRTC applications. -These constraints provide powerful tools for adapting media quality dynamically in response to changing network conditions or device capabilities. -By implementing these techniques, you can: - -- Improve user experience on limited bandwidth connections -- Reduce data consumption for users with data caps -- Optimize performance on lower-powered devices -- Ensure more consistent connection quality across various network conditions - -You can use this code as a starting point for your own applications that need to control media quality and bandwidth usage. \ No newline at end of file diff --git a/docs/guide/media/media-devices.md b/docs/guide/media/media-devices.md index a7752531..515f44f2 100644 --- a/docs/guide/media/media-devices.md +++ b/docs/guide/media/media-devices.md @@ -173,10 +173,7 @@ MediaDevices.addDeviceChangeListener(listener); MediaDevices.removeDeviceChangeListener(listener); ``` -## Conclusion - -This guide has provided a comprehensive overview of working with media devices in the WebRTC library. -When implementing media device handling in your applications, consider these best practices: +## Best Practices - Always check if devices exist before trying to use them - Handle the case where no devices are available gracefully @@ -184,5 +181,3 @@ When implementing media device handling in your applications, consider these bes - Implement hotplug listeners to dynamically update available devices - Remember to unregister device listeners when they're no longer needed - Query devices on worker threads, not on UI or render threads. - -You can use the code examples in this guide as a starting point for applications that need to work with media devices in WebRTC scenarios like video conferencing, live streaming, or media recording. \ No newline at end of file diff --git a/docs/guide/monitoring/logging.md b/docs/guide/monitoring/logging.md index 664db325..a2f80f78 100644 --- a/docs/guide/monitoring/logging.md +++ b/docs/guide/monitoring/logging.md @@ -128,6 +128,3 @@ Logging.addLogSink(Logging.Severity.VERBOSE, new Slf4jLogSink()); ``` This approach allows you to integrate WebRTC's native logging with your application's existing logging infrastructure. - -## Conclusion -By using the provided methods and custom log sinks, you can effectively capture, process, and manage log messages to aid in debugging and monitoring your application. Whether you choose to use the native logging capabilities or integrate with existing logging frameworks, the WebRTC logging system is designed to be adaptable to your needs. diff --git a/docs/guide/monitoring/rtc-stats.md b/docs/guide/monitoring/rtc-stats.md index 760f6686..bf44c885 100644 --- a/docs/guide/monitoring/rtc-stats.md +++ b/docs/guide/monitoring/rtc-stats.md @@ -391,7 +391,3 @@ Different stat types have different attributes. Here are some common attributes - framesPerSecond (video) Remember that the available attributes may vary depending on the state of the connection. - -## Conclusion - -WebRTC statistics provide essential insights into the performance and health of your real-time communications. By leveraging the RTC Stats API in webrtc-java, you can monitor connection quality, diagnose issues, and optimize your application's performance. The ability to collect detailed metrics on packets, jitter, latency, and more allows you to make data-driven decisions. \ No newline at end of file diff --git a/docs/guide/video/camera-capture.md b/docs/guide/video/camera-capture.md index a1b8d723..3d872aa7 100644 --- a/docs/guide/video/camera-capture.md +++ b/docs/guide/video/camera-capture.md @@ -386,9 +386,3 @@ When implementing camera capture in your application, consider these best practi 5. **Properly manage resources**: Always stop and dispose of video sources when they're no longer needed. 6. **Handle device changes**: Implement device change listeners to handle cameras being connected or disconnected. 7. **Consider privacy concerns**: In some applications, you may need to request permission to use the camera. - -## Conclusion - -This guide has demonstrated how to set up a WebRTC peer connection with camera video capture capabilities and how to receive and process both local and remote video frames. - -Camera capture is essential for video conferencing applications, video chat, remote interviews, and any other applications where face-to-face communication is important. \ No newline at end of file diff --git a/docs/guide/video/custom-video-source.md b/docs/guide/video/custom-video-source.md index e88f6fce..4a97ab1e 100644 --- a/docs/guide/video/custom-video-source.md +++ b/docs/guide/video/custom-video-source.md @@ -226,9 +226,3 @@ videoSource.dispose(); // If you're using a scheduled executor for pushing frames videoStreamer.stop(); ``` - -## Conclusion - -The `CustomVideoSource` provides a flexible way to integrate external video sources with WebRTC. By understanding the video format parameters and properly managing the video frame flow, you can create applications that use custom video from virtually any source. - -For more information on video processing and other WebRTC features, refer to the additional guides in the documentation. \ No newline at end of file diff --git a/docs/guide/video/desktop-capture.md b/docs/guide/video/desktop-capture.md index 92317c17..b9616c52 100644 --- a/docs/guide/video/desktop-capture.md +++ b/docs/guide/video/desktop-capture.md @@ -194,18 +194,13 @@ Without that permission, the windows of other apps are listed without their titl While a captured window is minimized or hidden, its last image repeats. When it is closed, capture stops. -## Conclusion - -This guide has demonstrated how to set up a WebRTC peer connection with desktop video capture capabilities. -When implementing desktop capture in your application, remember to: +## Best Practices - Always dispose of resources properly to prevent memory leaks - Periodically refresh source lists to handle dynamic changes in available screens and windows - Consider the performance implications of higher resolutions and frame rates - Handle potential permission requirements on different operating systems -Desktop capture is particularly useful for screen sharing applications, remote assistance tools, collaborative workspaces, and educational platforms where visual content sharing is essential. - ## Sharing system audio Add a [`SystemAudioSource`](/guide/audio/system-audio-source) audio track to share