Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
37 commits
Select commit Hold shift + click to select a range
2b76bf3
docs: write the WHIP and WHEP page in shorter sentences
devopvoid Nov 12, 2026
aa38375
docs: write the media files page in shorter sentences
devopvoid Nov 12, 2026
ab58e42
docs: write the end-to-end encryption page in shorter sentences
devopvoid Nov 12, 2026
3cfb816
docs: write the video codecs page in shorter sentences
devopvoid Nov 12, 2026
0c001df
docs: write the media recording page in shorter sentences
devopvoid Nov 12, 2026
471bc70
docs: write the statistics page in shorter sentences
devopvoid Nov 13, 2026
f1386d3
docs: write the sending media page in shorter sentences
devopvoid Nov 13, 2026
c99a965
docs: write the peers and signaling page in shorter sentences
devopvoid Nov 13, 2026
1c5d8af
docs: write the encoded transforms page in shorter sentences
devopvoid Nov 13, 2026
df1a3c1
docs: write the core API page in shorter sentences
devopvoid Nov 13, 2026
b038b1c
docs: write the threads page in shorter sentences
devopvoid Nov 13, 2026
b6011b0
docs: write the receiving media page in shorter sentences
devopvoid Nov 13, 2026
c373e40
docs: write the client transforms page in shorter sentences
devopvoid Nov 13, 2026
b033a35
docs: write the client API overview in shorter sentences
devopvoid Nov 13, 2026
4f6049f
docs: drop the outdated warning on injected audio
devopvoid Nov 13, 2026
bc79949
docs: write the system audio source page in shorter sentences
devopvoid Nov 13, 2026
552a99b
docs: write the media constraints page in shorter sentences
devopvoid Nov 13, 2026
dad4e12
docs: write the custom video source page in shorter sentences
devopvoid Nov 13, 2026
1e5b0a4
docs: write the native image page in shorter sentences
devopvoid Nov 13, 2026
4efc0b5
docs: write the peer connection configuration page in shorter sentences
devopvoid Nov 13, 2026
382dbee
docs: write the camera video page in shorter sentences
devopvoid Nov 13, 2026
e457f3e
docs: write the custom audio source page in shorter sentences
devopvoid Nov 13, 2026
f2cb3ca
docs: write the headless audio page in shorter sentences
devopvoid Nov 13, 2026
c11f994
docs: write the file transfer page in shorter sentences
devopvoid Nov 13, 2026
92c5a2c
docs: write the release notes of the build page in shorter sentences
devopvoid Nov 13, 2026
9b1e1d7
docs: split a long paragraph of the examples page
devopvoid Nov 13, 2026
623c8f3
docs: write the speech detection rules of audio processing as a list
devopvoid Nov 13, 2026
033e44c
docs: write the client data channels page in shorter sentences
devopvoid Nov 13, 2026
3b771a8
docs: split a long paragraph of the core data channels page
devopvoid Nov 13, 2026
acdb8a9
docs: split the introduction of the media devices page
devopvoid Nov 13, 2026
58a6c4e
docs: split a long sentence of the field trials page
devopvoid Nov 13, 2026
0fa618f
docs: split the macOS paragraph of the desktop video page
devopvoid Nov 13, 2026
d4669cd
docs: write the adapter notes of voice activity detection as a list
devopvoid Nov 13, 2026
b41646b
docs: split a long sentence of the video buffer converter page
devopvoid Nov 13, 2026
6180387
docs: split a paragraph of the sending media page
devopvoid Nov 13, 2026
8db7cd5
docs: correct device query failures on the client API design page
devopvoid Nov 13, 2026
36bb577
docs: drop the conclusion sections
devopvoid Nov 13, 2026
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
41 changes: 25 additions & 16 deletions docs/guide/advanced/encoded-transforms.md
Original file line number Diff line number Diff line change
@@ -1,17 +1,10 @@
# Encoded Transforms

This guide explains how to read and change the encoded frames of a call as they pass through an `RTCRtpSender` or an `RTCRtpReceiver`, the way [WebRTC Encoded Transforms](https://www.w3.org/TR/webrtc-encoded-transform/) (also known as insertable streams) do in a browser. It covers:
This guide explains how to read and change the encoded frames of a call as they pass through an `RTCRtpSender` or an `RTCRtpReceiver`. It works the way [WebRTC Encoded Transforms](https://www.w3.org/TR/webrtc-encoded-transform/), also known as insertable streams, do in a browser.

- Setting a transform on a sender or a receiver
- Reading and replacing a frame's payload
- Dropping frames
- What each frame tells about itself
- Asking for key frames
- What end-to-end encryption needs of a transform
- Threading and performance
- Sending frames that are already encoded through an injector
A sender's transform sees every frame after the encoder and before the packetizer. A receiver's transform sees every frame after the depacketizer and before the decoder. Everything between the two, whether that is the network, a TURN server or an SFU, only ever carries what the transform produced.

A sender's transform sees every frame after the encoder and before the packetizer; a receiver's sees every frame after the depacketizer and before the decoder. Everything between the two, whether that is the network, a TURN server or an SFU, only ever carries what the transform produced. That is what makes end-to-end encryption possible, and also frame metadata, watermarks, or analysis of the encoded stream.
That is what makes end-to-end encryption possible, and also frame metadata, watermarks, or analysis of the encoded stream.

## Setting a Transform

Expand Down Expand Up @@ -39,7 +32,11 @@ for (RTCRtpReceiver receiver : peerConnection.getReceivers()) {
Set the transform of a video sender before negotiating when you can. The first transform set on a running video sender restarts its send stream, which costs a key frame. Later changes, including removing it, cost nothing.
:::

The transform belongs to the native sender or receiver, not to the Java object: every `RTCRtpSender` instance for the same sender shares it, setting it through one replaces what was set through another, and disposing of an instance leaves it in place.
The transform belongs to the native sender or receiver, not to the Java object:

- Every `RTCRtpSender` instance for the same sender shares it.
- Setting it through one instance replaces what was set through another.
- Disposing of an instance leaves it in place.

### With the Client API

Expand Down Expand Up @@ -71,7 +68,9 @@ receiver.setTransform(frame -> {
});
```

`setData` takes a `ByteBuffer` (its remaining bytes) or a `byte[]`, and the new payload may be larger or smaller than the old one. The payload is only copied into Java when `getData()` is called, so a transform that looks at the metadata alone costs no copy at all. Each transform thread reuses at most 256 KiB of direct scratch memory. Larger payloads use a temporary buffer sized to the frame, released for garbage collection when the transform returns. Workloads with frequent larger frames therefore allocate a buffer for each of them.
`setData` takes a `ByteBuffer` (its remaining bytes) or a `byte[]`, and the new payload may be larger or smaller than the old one. The payload is only copied into Java when `getData()` is called, so a transform that looks at the metadata alone costs no copy at all.

Each transform thread reuses at most 256 KiB of direct scratch memory. Larger payloads use a temporary buffer sized to the frame, released for garbage collection when the transform returns. Workloads with frequent larger frames therefore allocate a buffer for each of them.

A frame, and the buffer `getData()` returned, is valid only while `transform` runs and only on its thread. The buffer's memory is reused for the next frame, so copy what you need to keep. Reading or changing a frame after the transform returned throws an `IllegalStateException`; its metadata stays readable.

Expand Down Expand Up @@ -105,10 +104,16 @@ Both do nothing for audio, and `generateKeyFrame()` nothing before the connectio

## End-to-End Encryption

[End-to-End Encryption](/guide/advanced/end-to-end-encryption) builds on these transforms: a `KeyProvider` makes the encryptors and decryptors, in the SFrame format of RFC 9605, and takes care of the keys of every participant, key rotation, replay protection and frames that fail. Use it rather than a transform of your own, which would have to settle all of that, and a frame format browsers can implement too. Two points matter for any encrypting transform that is written anyway:
[End-to-End Encryption](/guide/advanced/end-to-end-encryption) builds on these transforms. A `KeyProvider` makes the encryptors and decryptors, in the SFrame format of RFC 9605. It also takes care of the keys of every participant, key rotation, replay protection and frames that fail.

Use it rather than a transform of your own, which would have to settle all of that, and a frame format that browsers can implement too. Two points matter for any encrypting transform that is written anyway:

- **Keep the codec header in the clear.** The packetizer and depacketizer read the start of some frames: a VP8 receiver finds key frames and the frame size in the first bytes, and the AV1 packetizer splits frames into their OBUs. Browsers leave 10 bytes of a VP8 key frame, 3 of a VP8 delta frame and 1 of an Opus frame unencrypted, and authenticate them as associated data instead. H.264 is split at its NAL unit start codes before it is sent, so encrypting it whole breaks the packetizer.
- **Make each frame self-contained.** Frames are lost, so each needs what it takes to be decrypted, such as its nonce, and a receiver must drop a frame that fails to authenticate rather than pass it on.
- **Keep the codec header in the clear.** The packetizer and depacketizer read the start of some frames:
- A VP8 receiver finds key frames and the frame size in the first bytes.
- The AV1 packetizer splits frames into their OBUs.
- H.264 is split at its NAL unit start codes before it is sent, so encrypting it whole breaks the packetizer.

Browsers leave 10 bytes of a VP8 key frame, 3 of a VP8 delta frame and 1 of an Opus frame unencrypted, and authenticate them as associated data instead.

A transform that protects what passes through it can keep observers of the encoded frames, such as a [`MediaRecorder`](/guide/media/media-recording), from seeing it in the clear by returning false from `allowsPlaintextObservers()`.

Expand Down Expand Up @@ -148,6 +153,10 @@ RTCEncodedVideoFrameInjector injector = transceiver.getSender()
injector.injectFrame(frameData, rtpTimestamp, keyFrame, width, height);
```

The frames must be in the codec the sender negotiated, which `setCodecPreferences` on the transceiver can make sure of, and the RTP timestamp runs on the 90 kHz video clock. VP9 cannot be injected: WebRTC sends injected frames without the scalability structure a VP9 receiver needs with every key frame, so the receiver drops them. H.264, H.265 and AV1 frames arrive. An audio sender takes an `RTCEncodedAudioFrameInjector` the same way, whose frames carry the payload type of their codec, from the sender's `getParameters().codecs`. Do not inject audio through a factory that sends audio captured by its audio device module, as WebRTC feeds captured audio to every audio sender.
The frames must be in the codec the sender negotiated, which `setCodecPreferences` on the transceiver can make sure of. The RTP timestamp runs on the 90 kHz video clock.

VP9 cannot be injected: WebRTC sends injected frames without the scalability structure that a VP9 receiver needs with every key frame, so the receiver drops them. H.264, H.265 and AV1 frames arrive.

An audio sender takes an `RTCEncodedAudioFrameInjector` the same way. Its frames carry the payload type of their codec, from the sender's `getParameters().codecs`.

Giving the sender a track through `replaceTrack` ends the injection, and `dispose()` releases the injector. The observer runs on a callback thread shared with other WebRTC observers, so it may call back into WebRTC but should not block.
Loading
Loading