Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
21 changes: 21 additions & 0 deletions docs/guide/media/media-files.md
Original file line number Diff line number Diff line change
Expand Up @@ -351,6 +351,27 @@ A live stream has no end and no length:
- Pausing stops reading from the server, but the server keeps sending. Resuming carries on from where playback paused, which leaves it behind real time by as long as it was paused.
- To get back to live, close the source and open it again.

### Video Size of a Camera

Many cameras say how large their pictures are only with each keyframe. FFmpeg reads 5 seconds of a source when it opens it. If the first keyframe comes later, `MediaInfo` reports a video size of 0x0. The video plays all the same, and its frames have their size.

To know the size when the source opens, let FFmpeg read for longer than the camera's keyframe interval:

```java
// Import required classes
import dev.onvoid.webrtc.media.player.MediaReader;
import dev.onvoid.webrtc.media.player.MediaReaderOptions;

MediaReaderOptions options = MediaReaderOptions.builder()
.analyzeDuration(Duration.ofSeconds(15))
.build();

MediaReader reader = new MediaReader("rtsp://camera.local/stream1", options);
MediaFileSource source = new MediaFileSource(reader, false);
```

The source takes over the reader and closes it. Opening takes up to the analysis duration on top of the timeout, as the camera sends in real time. FFmpeg also stops after 5 MB. For a stream of a high bit rate, raise that limit with `probeSize(bytes)` as well.

## Closing

Closing a `MediaFileSource` stops playback, and releases the player along with both media sources. Before that, release the senders a peer connection handed out, and the tracks:
Expand Down
4 changes: 2 additions & 2 deletions webrtc-media/src/main/cpp/include/JNI_MediaReader.h
Original file line number Diff line number Diff line change
Expand Up @@ -25,10 +25,10 @@ extern "C" {
/*
* Class: dev_onvoid_webrtc_media_player_MediaReader
* Method: open
* Signature: (Ljava/lang/String;J)J
* Signature: (Ljava/lang/String;JJJ)J
*/
JNIEXPORT jlong JNICALL Java_dev_onvoid_webrtc_media_player_MediaReader_open
(JNIEnv *, jclass, jstring, jlong);
(JNIEnv *, jclass, jstring, jlong, jlong, jlong);

/*
* Class: dev_onvoid_webrtc_media_player_MediaReader
Expand Down
16 changes: 14 additions & 2 deletions webrtc-media/src/main/cpp/include/media/MediaReader.h
Original file line number Diff line number Diff line change
Expand Up @@ -50,6 +50,13 @@ namespace ffmpeg
MediaReader(const MediaReader &) = delete;
MediaReader & operator=(const MediaReader &) = delete;

// How much of the source libavformat reads when it opens it, to
// find what the streams hold: in microseconds of the source and
// in bytes. 0 leaves its default. Opening may take the analysis
// duration on top of the timeout, which a live stream needs to
// send that much. Call this before Open.
void SetAnalysis(int64_t analyze_duration_us, int64_t probe_size);

// Opens the given source and selects the video and audio stream
// that are meant to be played. Returns 0, or the negative AVERROR
// libavformat reported, in which case the reader stays closed.
Expand Down Expand Up @@ -134,8 +141,9 @@ namespace ffmpeg
// makes the blocking call fail with AVERROR_EXIT.
static int OnInterrupt(void * opaque);

// Starts the time the next blocking call has, and ends it.
void BeginBlocking();
// Starts the time the next blocking call has, the timeout and the
// given extra time, and ends it.
void BeginBlocking(int64_t extra_us = 0);
void EndBlocking();

// A blocking call that failed because it ran out of time rather
Expand All @@ -149,6 +157,10 @@ namespace ffmpeg

const int64_t timeout_us_;

// 0 for the defaults of libavformat.
int64_t analyze_duration_us_ = 0;
int64_t probe_size_ = 0;

// When the current blocking call runs out of time, on the clock
// of av_gettime_relative(), or 0 while none is running.
std::atomic<int64_t> deadline_us_{ 0 };
Expand Down
9 changes: 9 additions & 0 deletions webrtc-media/src/main/cpp/include/media/VideoDecoder.h
Original file line number Diff line number Diff line change
Expand Up @@ -177,6 +177,11 @@ namespace ffmpeg
// for a key frame, which it is after it lost pictures it needs.
bool SkipUntilKeyFrame(const AVPacket * packet);

// Whether an error of the codec only says that the stream started
// between key frames: one before the first picture, which is
// passed over.
bool IsLeadingError(int result) const;

void KeepPacket(const AVPacket * packet);
void DropKeptPackets();
void DropPending();
Expand Down Expand Up @@ -233,6 +238,10 @@ namespace ffmpeg
// and starts again at a key frame.
bool awaiting_key_frame_ = false;
size_t skipped_ = 0;

// Whether the codec has given out a picture since the decoder
// was opened.
bool has_pictures_ = false;
std::deque<Pending> pending_;

// Where the frames after a seek are to start.
Expand Down
6 changes: 5 additions & 1 deletion webrtc-media/src/main/cpp/src/JNI_MediaReader.cpp
Original file line number Diff line number Diff line change
Expand Up @@ -62,7 +62,8 @@ namespace
}

JNIEXPORT jlong JNICALL Java_dev_onvoid_webrtc_media_player_MediaReader_open
(JNIEnv * env, jclass caller, jstring source, jlong timeoutUs)
(JNIEnv * env, jclass caller, jstring source, jlong timeoutUs, jlong analyzeDurationUs,
jlong probeSize)
{
return ffmpeg::CallFromJava<jlong>(env, 0, [&]() -> jlong {
if (source == nullptr) {
Expand Down Expand Up @@ -92,6 +93,9 @@ JNIEXPORT jlong JNICALL Java_dev_onvoid_webrtc_media_player_MediaReader_open
}

auto reader = std::make_unique<ffmpeg::MediaReader>(static_cast<int64_t>(timeoutUs));

reader->SetAnalysis(static_cast<int64_t>(analyzeDurationUs), static_cast<int64_t>(probeSize));

int result = reader->Open(url);

if (result < 0) {
Expand Down
24 changes: 21 additions & 3 deletions webrtc-media/src/main/cpp/src/media/MediaReader.cpp
Original file line number Diff line number Diff line change
Expand Up @@ -57,6 +57,12 @@ namespace ffmpeg
Close();
}

void MediaReader::SetAnalysis(int64_t analyze_duration_us, int64_t probe_size)
{
analyze_duration_us_ = std::max<int64_t>(analyze_duration_us, 0);
probe_size_ = std::max<int64_t>(probe_size, 0);
}

int MediaReader::Open(const std::string & url)
{
Close();
Expand All @@ -76,6 +82,15 @@ namespace ffmpeg
format_context_->interrupt_callback.callback = &MediaReader::OnInterrupt;
format_context_->interrupt_callback.opaque = this;

// Set on the context rather than passed as options, which only fail
// where an allocation can. 0 is what libavformat takes for its own
// choice of analysis duration; its probe size has a default instead.
format_context_->max_analyze_duration = analyze_duration_us_;

if (probe_size_ > 0) {
format_context_->probesize = probe_size_;
}

AVDictionary * options = nullptr;

int result = AllocationFault::CheckOption(
Expand Down Expand Up @@ -105,7 +120,9 @@ namespace ffmpeg
return TranslateError(result);
}

BeginBlocking();
// A live stream takes as long to send what is analyzed as it lasts,
// which the timeout alone may not cover.
BeginBlocking(analyze_duration_us_);
result = avformat_find_stream_info(format_context_, nullptr);
EndBlocking();

Expand Down Expand Up @@ -382,7 +399,7 @@ namespace ffmpeg
return 0;
}

void MediaReader::BeginBlocking()
void MediaReader::BeginBlocking(int64_t extra_us)
{
timed_out_.store(false);

Expand All @@ -394,8 +411,9 @@ namespace ffmpeg
// Saturated: a timeout too long to count arrives as INT64_MAX, and the
// sum would wrap around into a deadline that has passed already.
const int64_t now = av_gettime_relative();
const int64_t time_us = extra_us > INT64_MAX - timeout_us_ ? INT64_MAX : timeout_us_ + extra_us;

deadline_us_.store(timeout_us_ > INT64_MAX - now ? INT64_MAX : now + timeout_us_);
deadline_us_.store(time_us > INT64_MAX - now ? INT64_MAX : now + time_us);
}

void MediaReader::EndBlocking()
Expand Down
26 changes: 26 additions & 0 deletions webrtc-media/src/main/cpp/src/media/VideoDecoder.cpp
Original file line number Diff line number Diff line change
Expand Up @@ -342,6 +342,7 @@ namespace ffmpeg
draining_ = false;
awaiting_key_frame_ = false;
skipped_ = 0;
has_pictures_ = false;
discarding_ = false;
time_base_ = { 0, 1 };
start_time_us_ = 0;
Expand Down Expand Up @@ -439,9 +440,25 @@ namespace ffmpeg
}
}

if (packet != nullptr && IsLeadingError(result)) {
// The packet is taken all the same: the codec has it, or it is
// one of those that cannot be decoded.
return 0;
}

return result;
}

bool VideoDecoder::IsLeadingError(int result) const
{
// A live stream joined between key frames, as an IP camera that sends
// its parameter sets only in band is: what comes before the first key
// frame cannot be decoded, and is no fault of the stream. Passing
// over it starts decoding at that key frame. Once there are pictures,
// such an error is a fault.
return result == AVERROR_INVALIDDATA && !has_pictures_;
}

int VideoDecoder::ReceiveFrame(AVFrame ** frame, int64_t * timestamp_us)
{
if (codec_context_ == nullptr || decoded_ == nullptr) {
Expand Down Expand Up @@ -479,9 +496,18 @@ namespace ffmpeg

int result = ReceivePicture(frame, timestamp_us);

if (IsLeadingError(result)) {
// With frame threads the codec tells of a packet it could not
// decode on the way out, and the pictures behind it are still
// to come.
continue;
}
if (result < 0) {
return result;
}

has_pictures_ = true;

if (*frame == nullptr) {
// Dropped before it was read back and converted.
dropped_before_ = true;
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -19,6 +19,7 @@
import java.io.IOException;
import java.nio.file.Path;
import java.time.Duration;
import java.util.Objects;
import java.util.concurrent.atomic.AtomicBoolean;
import java.util.function.Supplier;

Expand Down Expand Up @@ -162,6 +163,27 @@ public MediaFileSource(String source, Duration timeout, boolean hardwareDecoding
hardwareDecoding);
}

/**
* Plays the source of a reader opened by the caller, such as one given
* {@link MediaReaderOptions}, decoding its video in hardware where the
* platform can if asked to, as {@link MediaPlayer#MediaPlayer(MediaReader,
* CustomVideoSource, CustomAudioSource, boolean)} describes.
* <p>
* The reader is taken over, and closed with this source, or right away
* if this source cannot be made. It must not be used afterwards.
*
* @param reader An open reader of the source to play.
* @param hardwareDecoding Whether to decode video in hardware if possible.
* HEVC is decoded in hardware regardless.
*
* @throws IOException if the source cannot be decoded.
* @throws IllegalStateException if the reader is closed.
*/
public MediaFileSource(MediaReader reader, boolean hardwareDecoding) throws IOException {
this(Objects.requireNonNull(reader, "Reader is null"), CustomVideoSource::new,
CustomAudioSource::new, hardwareDecoding);
}

/**
* Builds on a reader that is open, which this source takes over whether
* or not it is made. The suppliers make the media sources, which tests
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -23,6 +23,12 @@
* A stream that is absent is reported as a zero size, frame rate, sample rate
* or channel count, so {@link #hasVideo()} and {@link #hasAudio()} are the
* questions to ask before reading the rest.
* <p>
* A video stream can be there with a zero size, too: a live stream that
* carries its parameter sets only in band, as many IP cameras do, says how
* large its pictures are only with its first keyframe. If that comes later
* than FFmpeg looks for it when the source is opened, the size is not known
* yet. The video plays all the same, and its frames have their size.
*
* @author Alex Andres
*/
Expand All @@ -31,10 +37,10 @@ public class MediaInfo {
/** How long the source runs in microseconds, or 0 if it is not known. */
private final long durationUs;

/** The width of the video as it is shown, or 0 if there is no video. */
/** The width of the video as it is shown, or 0 if there is none or it is not known. */
private final int videoWidth;

/** The height of the video as it is shown, or 0 if there is no video. */
/** The height of the video as it is shown, or 0 if there is none or it is not known. */
private final int videoHeight;

/** The frame rate of the video, or 0 if there is no video. */
Expand Down Expand Up @@ -90,20 +96,25 @@ public long getDurationUs() {
}

/**
* Returns whether the source has a video stream that can be played.
* Returns whether the source has a video stream that can be played. Its
* size may not be known yet, see the class description.
*
* @return True if there is video.
*/
public boolean hasVideo() {
return videoWidth > 0 && videoHeight > 0;
// The codec, not the size: native code names one for every video
// stream it plays, while the size can be missing until the first
// keyframe.
return videoCodec != null;
}

/**
* Returns the width of the video as it is shown, in square pixels and
* turned upright. A video stored sideways reports its stored height here,
* while its frames keep their stored size and carry the rotation.
*
* @return The width, or {@code 0} if there is no video.
* @return The width, or {@code 0} if there is no video or its size is not
* known yet.
*/
public int getVideoWidth() {
return videoWidth;
Expand All @@ -114,7 +125,8 @@ public int getVideoWidth() {
* turned upright. A video stored sideways reports its stored width here,
* while its frames keep their stored size and carry the rotation.
*
* @return The height, or {@code 0} if there is no video.
* @return The height, or {@code 0} if there is no video or its size is not
* known yet.
*/
public int getVideoHeight() {
return videoHeight;
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -19,6 +19,7 @@
import java.io.IOException;
import java.nio.file.Path;
import java.time.Duration;
import java.util.Objects;

import dev.onvoid.webrtc.internal.Timeouts;

Expand Down Expand Up @@ -96,11 +97,30 @@ public MediaReader(String source) throws IOException {
* @throws IllegalArgumentException if the timeout is negative.
*/
public MediaReader(String source, Duration timeout) throws IOException {
this(source, MediaReaderOptions.builder().timeout(timeout).build());
}

/**
* Opens the given media source with the given options: the timeout of
* each operation on it, and how much of it FFmpeg reads to find out what
* its streams hold.
*
* @param source The path or {@code rtsp://} URL of the source to open.
* @param options How to open the source.
*
* @throws IOException if the source cannot be opened in time, or holds
* nothing that can be played.
*/
public MediaReader(String source, MediaReaderOptions options) throws IOException {
Objects.requireNonNull(options, "Options are null");

// Zero is the one timeout without limit: a positive one rounds up to a
// microsecond, and one too long to count is as long as can be counted.
Timeouts.requireNonNegative(timeout, "timeout");
Duration analyzeDuration = options.getAnalyzeDuration();

handle = open(source, Timeouts.toMicros(timeout));
handle = open(source, Timeouts.toMicros(options.getTimeout()),
analyzeDuration != null ? Timeouts.toMicros(analyzeDuration) : 0,
options.getProbeSize());
}

/**
Expand Down Expand Up @@ -149,7 +169,8 @@ synchronized long detach() {
return detaching;
}

private static native long open(String source, long timeoutUs) throws IOException;
private static native long open(String source, long timeoutUs, long analyzeDurationUs,
long probeSize) throws IOException;

private static native MediaInfo info(long handle);

Expand Down
Loading
Loading