Repository navigation
Warn of an exiftool_runtime pattern track far longer than the video - #908
Merged
Merged
Conversation
A file named on the command line that is neither an image nor a video
that mapillary_tools can process was left out of the description with
only a warning, "Skipping ..., which is not a supported image or video",
and process exited 0. Named on its own, such a file gave an empty
description and a successful run, so a user who named the original of an
Insta360 camera (.insv), which has to be stitched first, saw nothing
uploaded and no error.
Fail such a file with an error entry of its own, of the new type
MapillaryUnsupportedFileError: "Not a supported image or video". For an
Insta360 original, the message goes on to say to stitch it with Insta360
Studio first, and then to geotag the stitched video by the original with
--video_geotag_source '{"source": "exiftool_runtime", "pattern":
"../%g.insv"}', as the README does. Like any error, it makes process
exit 6, or 0 with --skip_process_errors, and upload skips it. The entry
names no file type, as the file is of none; the process summary counts
it as an "other file". A new error type rather than an existing one, as
none says what is wrong: the geotagging errors would let another source
try the file, and the file is not a video.
Unchanged: the files in a named folder that cannot be processed are
still left alone without a word, a named zip file is still uploaded as
it is by process_and_upload, and the process report still lists the
unsupported files, named or found, as unsupported_format.
Checked process on named images, videos, zip files, text files and an
Insta360 original, alone, together, named twice, beside a named folder,
with --filetypes image, and on folders holding such files, and
video_process and sample_video: only the runs that named an unsupported
file change, each by the one error entry. process_and_upload of named
zip files with a named text file uploads the zip files and fails the
text file.
Interval sampling cuts a frame every N seconds of the whole video, whatever GPS track will geotag the frames, so the frames past either end of the track were written and then failed geotagging, which failed the command: the last frame of a BlackVue video, cut after the last GPS point, with "Unable to extract GPS Longitude or GPS Latitude from the image", or the last frame of a GoPro MAX 2 timelapse sampled every 0.25 s against its GPX, 0.386 s past the end of the GPX, with MapillaryOutsideGPXTrackError. At a 1 s interval, 17 of 49 sample videos had such frames: one or two at an end of most, and up to 64 at the start of those whose GPS starts late. Leave such frames out, as distance sampling leaves out the frames outside the track it samples along, and warn once per video, e.g. "2 of the 61 samples of X fall outside its GPS track (A to B) and were left out". The track is the one that will geotag the frames: the video's own track, which sampling now writes into the frames, or a GPX or NMEA track of --geotag_source, which the geotagging stage locates the frames along by their capture times, with its offset, its start time and its gaps of more than an hour, as it does. If the track covers none of the frames, they are all written as before, so that a video that misses its track fails rather than goes without a word. The frames before the start of the track are kept where leaving them out would move the others: the legacy gopro, blackvue and camm sources and exiftool_xml geotag a frame by the time since the first frame of its video, which they place at the start of the video, and so does a GPX with --interpolation_use_gpx_start_time, which also starts at the first frame of all the videos sampled, so that with several videos none is left out. The sample_video command cannot tell which sources will geotag its frames, so it keeps them too. Nothing is left out of a GPX track that another source, which geotags frames by the GPS of their video or by another track file, may geotag instead. On those 49 videos, sampled every second with the default sources, the 17 lose only the frames that failed, and 16 of them now exit 0 (the other fails as before for an unrelated reason); every other description entry is the same. The BlackVue gives 60 frames and the GoPro MAX 2 timelapse 4, both exit 0. With the legacy sources, the frames before a late GPS start are kept and geotagged as before, and with a GPX of --geotag_source only the frames that it rejected are left out. Distance sampling does not change.
process uploads the track it places a GPX on a video by with the video, and the frames are placed along it. Since mapillary#874, a time in a gap of more than MAPILLARY_TOOLS_MAX_GPX_GAP_SECONDS (an hour) between two fixes is outside the track, but the track clipped to the video was still interpolated across such a gap. A video that starts in a gap got a point at video time 0 interpolated between the last fix before the gap and the first one after it. On a GoPro MAX video with a GPX whose fixes resume 99 seconds into the video, 2 hours after the previous one 5 km away, that point was 151 m from where the full track puts the start of the video, and the first 99 seconds of frames were placed between the two. At driving speed it is kilometres off. When video time 0, or the end of the video, falls in such a gap, leave out the fixes on its far side, and interpolate no point across it: the track starts (or ends) at the first fix past the gap, as one that starts after the video does, and process still warns that it covers only part of the video. A gap inside the video fails it with MapillaryOutsideGPXTrackError, which names the gap and suggests sampling the video with video_process and --video_geotag_source gpx (or nmea), which leaves out the frames in the gap. So does a gap that the video falls in but for an instant at one end, such as the gap after a lone first fix that the track is started with the video at. A track assumed to start with a video without a GPS clock keeps its end, which may be that of a timelapse, unless the end of the video at 1x falls in a gap. The limit is in GPS seconds, so it spans less video in a timelapse. Sampling is unchanged. Checked process with GPX and NMEA tracks made to have gaps: - The GoPro MAX video above: its track starts at 99.085 s, with the warning; with --interpolation_offset_time 2, at 97.085 s. With the track ending 159 s into the video and resuming 2 hours later, it ends at 159.085 s. With both, it runs from 99.085 to 159.085 s. - A gap of 70 s inside the video, with the limit at 60 s, fails it, by a GPS clock or with --interpolation_use_gpx_start_time, without a GPS clock, from an NMEA file, and in a timelapse, whose limit of 30 s is in GPS time. With native as the next source, native geotags it. video_process samples the same videos and tracks as before. - Without a GPS clock, a track that resumes 2 hours after the end of the video at 1x ends with the video, and one with a lone first fix 2 hours before the rest fails. With the limit raised above the gap, the tracks are interpolated across it as before. process with the GPX, NMEA and ExifTool tracks of sample videos and images from a dozen cameras (58 runs, among them GPX tracks of several trips with gaps of days, offsets and --interpolation_use_gpx_start_time), plain process, and video_process with GPX tracks write the same descriptions as before.
The legacy gopro_videos, blackvue_videos and camm sources geotag images from the videos they are given: the frames that sample_video took from them. An image that is not a frame of any of the videos got no entry at all: `process FRAMES --geotag_source blackvue_videos --geotag_source_path VIDEOS` left a foreign image in a folder of frames out of the description and exited 0. Since each video geotags only its own frames, nothing reported it; before, every video tried every image, and such an image was reported as outside the track. The legacy sources now fail such an image with "Not a frame of any video in VIDEOS". That is a geotagging error, which a later source can still replace: with `--geotag_source blackvue_videos --geotag_source exif` the image is geotagged from its EXIF, as before. And at the end of the source chain, a file that sources were let through but none of them described, nor said why, is failed with "The geotag sources ... gave no result for this image" (or video), so that every file given gets exactly one entry. Files that --filetypes leaves out are still left out: images by their names, and videos once a source finds them of other types, or the native reader cannot recognize one of the types asked for. On a folder of the frames of two BlackVue videos, with a foreign image among the frames and one beside them, the legacy sources fail the two images, and with exif after them geotag them as before. The frames, and process of a mixed folder with the default sources, exif, --filetypes image or gopro and --video_geotag_source native, of the frames with the default sources, of the frames of a GoPro and a CAMM video with their legacy sources, of an image sequence, and video_process with the default sources, the legacy source and exif, do not change.
A sentence cut short, such as the last one of a log when the recording stopped, could give a fix a wrong date. pynmea2 reads a sentence without its checksum, and the fields it can: an RMC sentence cut inside its date, "...,0.0,0.0,2212", reads as dated 2012-02-02. Since NMEA fixes are dated by the RMC sentences around them, the last fix of a log ending so took that date, and process placed a 10-minute video along a track that lasted 3976 days, with exit 0. A time cut short reads too: "00052" as 00:00:02. Treat such sentences as malformed, and skip them with the warning that malformed sentences get: - a sentence without a checksum, where the other sentences of the file have theirs: receivers end each sentence with one, so one without was cut short; - a time that is not hhmmss with an optional fraction of a second, and an RMC date that is not ddmmyy. A log whose sentences carry no checksum at all, as some converters write it, is read as before. Empty times and dates, as before a fix, are not malformed. On a 650-fix log that crosses midnight, written with each GGA sentence before its RMC sentence, cut inside the date of its last RMC sentence: the track is again 650 points over 649 s, with one warning naming the cut sentence. With 186 sentences cut short at random points, the 186 are skipped under one warning and no fix is misdated, where the track lasted 4006 days. Three other logs, one of them as a camera wrote it, give the same descriptions as before, for images and videos.
NMEA files are read as UTF-8 on every platform since geotagging videos
from them was added, so that a file that is not text fails as invalid.
That also failed a log for a single byte that is not UTF-8, such as one
damaged in a sentence: every image and video geotagged from it was
reported as "not a valid NMEA file". Where the default encoding was
Latin-1, as on Windows, such a file used to be read.
Decode a byte that is not text as U+FFFD instead. A sentence with one is
malformed, and skipped with the warning that malformed sentences get
("a byte that is not text"). Such a byte elsewhere, as in a header line
that some cameras write, is ignored. A file of which no fix can be read
and that has such bytes still fails as not a valid NMEA file, so a file
that is not text at all does too.
On a log of 222 sentences for a sequence of 113 images, with one byte
0xE9 inside a GGA sentence: 106 images are geotagged, as from the clean
log, where all 113 failed. Two of them, next to the lost fix, move by
1.4 m and 0.1 m. On a 650-fix log for a video, the video is geotagged
with one warning, where it failed. The other logs give the same
descriptions as before.
…ource Interval sampling with --video_geotag_source gpx or nmea geotagged none of its samples: each failed with "Unable to extract GPS Longitude or GPS Latitude from the image", and the command exited 6, on a GoPro MAX timelapse with its GPX and on a Sony video with the GPX or NMEA log of its GPS. Such a source geotags videos only, so the geotagging stage reads the samples' own positions, and interval sampling wrote positions only from the video's own GPS. Distance sampling with the same options worked, and so did interval sampling with --geotag_source gpx, which geotags the samples themselves. The message of mapillary#896, which located interval samples along the video's own GPS, said that they were placed along a GPX too; they were not. Locate each interval sample along the GPX or NMEA track as GPX sync placed it on the video, at the sample's video time, and write its position into the sample, as distance sampling does. Stamp it with the time of the track there, as distance sampling stamps the samples of a track from a file: for a video with a GPS clock, and without --interpolation_offset_time, that is the time it was stamped with before. With --interpolation_use_gpx_start_time, the track starts at the first sample, which ffmpeg cuts at time 0, as the samples of a video whose GPS was rejected as noise are stamped from it, not at the first frame, where GPX sync starts it for distance sampling. The samples outside the track, or in one of its gaps of more than an hour, are left out with a warning, as other samples outside their track are, unless a legacy source or exiftool_xml, which geotag samples by the GPS of their video, may geotag them instead. Distance sampling with such a source interpolated the track across its gaps: the frames in a gap of hours got positions between the fixes either side of it, which say nothing of where the camera was, and the geotagging stage read them from the frames. Since mapillary#874 a time in such a gap is outside the track, and the samples of a file of --geotag_source in it fail geotagging. Leave those frames out too, with one warning per video that names the gaps, so that distance and interval samples agree. The limit is in GPS seconds, as in the geotagging stage, so it spans less video in a timelapse. Distance sampling with a file of --geotag_source, which the geotagging stage geotags the frames by, or with a legacy source or exiftool_xml, is unchanged. On the GoPro MAX timelapse sampled every 0.5 s, the Sony video sampled every 20 s with its GPX and with its NMEA log, and a noisy GoPro MAX 2 timelapse with its GPX started with the video, every sample is geotagged, within 1.4 cm of the track at its capture time, and the commands exit 0 (the last sample of the GoPro MAX 2 timelapse, past the GPX, is left out). --geotag_source gpx and nmea, the video's own GPS and the legacy sources give the same descriptions as before. Checked distance sampling with tracks made to have gaps: - A GoPro MAX video with a GPX missing 70 s of fixes inside it, with MAPILLARY_TOOLS_MAX_GPX_GAP_SECONDS at 60: the 2098 frames in the gap are left out, and at 5 m 55 samples are written instead of 70, none in the gap, the 24 before it as before. The same with --interpolation_offset_time 2 and with --interpolation_use_gpx_start_time. At the default limit, nothing changes. - The same video with a GPX that resumes 99 s into it, 2 hours after its previous fix: the first 99 s are left out (46 samples instead of 59), where the track starts after the gap; with the GPX ending a minute later too, the 14 samples of that minute are written. Started with the video, the GPX covers only its first frame, which alone is written, instead of 45 samples placed along the gap. - A Sony video with a GPX and with an NMEA log missing 2 minutes of fixes, at a limit of 60 s: 257 samples at 10 m instead of 316, none in the gap, the first 114 as before. - A GoPro MAX timelapse with 41 s of GPS time missing, 1.4 s of video: at a limit of 30 s, its 40 frames are left out; at 60 s, nothing changes. - With these tracks as --geotag_source gpx or nmea, with the legacy gopro source next to --video_geotag_source gpx, and with interval sampling, the descriptions are the same as before. With the GPX and NMEA tracks of sample videos from a dozen cameras, among them GPX logs of several trips with gaps of more than an hour outside the videos (111 runs of video_process: --video_geotag_source gpx and nmea, with --interpolation_use_gpx_start_time or an offset, interval sampling, and --geotag_source gpx and nmea), and plain video_process by distance and by interval on five cameras, the descriptions are the same as before.
mapillary#889 reads the rate of a video's GPS clock as the least-squares slope through its timestamps, and keeps it at 1x within 5% or 2 seconds of drift. A few timestamps off the rest pull that slope far off on a short clip. Over 8 to 20 seconds of GoPro HERO7, GoPro MAX and PANOX V2 videos, timestamps a second or more off the line measure 0.24x to 1.33x, moving the ends of a clip by seconds; ten fixes over 9 seconds, the last of them 6 seconds late, measure 1.33x. Nor is a long clip safe: a clock that steps 18 seconds ahead two thirds into a 2-minute video measures 1.2x. And a few seconds of a PANOX V2 video can hold only 3 timestamps, one of them stamped seconds late, which read as a 3.5x or 4x timelapse. Measure the rate as the median of the slopes between pairs of timestamps (the Theil-Sen estimator), over an evenly spaced selection of 300 of them if there are more, which measures it to within 0.01% and reads a clock in a few milliseconds. Below the timelapse rate, keep it at 1x unless the timestamps span a minute or more, of video and of GPS time, on top of the 5% and 2-second rules. A timelapse must still show in the median step between neighbouring timestamps, now of at least two steps, and in both when there are only two. The anchor, the median of the timestamps mapped back to the start, and the cleaning of the timestamps are as before. On the 554 sample videos with a GPS clock: - Every video that records in real time keeps the same clock, at 1x. - The LabPano Pilot One and Era videos keep their rates, 1.1426 to 1.1429, and their clocks move by no more than 0.09 GPS seconds; their timestamps stay within 0.3 s of them, as before. One LabPano video, whose GPS drops out and comes back with its time several seconds behind, measures 1.136, near the rate of the others, rather than 1.093: three quarters of its timestamps now lie within a second of its clock, where a quarter did, and those before the dropout up to 6.6 s off it, where they were up to 4.2 s. - Timelapses keep their rates within 0.3%. A GoPro MAX timelapse whose first timestamp is late measures 14.98x rather than 14.93x, and a GoPro MAX 2 clip whose GPS is rejected as noise 12.5x rather than 12.39x; their samples move by at most 0.23 GPS seconds, 15 ms of video. Cut into 237,701 pieces of 2 to 45 seconds, the timestamps of these videos read a rate off 1x on 60 pieces of real-time videos (0.14x to 7x), and now on none. Timelapse pieces read as before, but for those of only two timestamps, from a timelapse stamped once per second of video: one step cannot tell a timelapse from a fix stamped seconds late. Samples of the GoPro HERO7, LabPano and GoPro MAX timelapse videos, sampled by interval and by distance, with and without a GPX made from their own tracks, change by the clock moves above and by no more: the HERO7's do not change.
Since --video_geotag_source gopro, camm and blackvue read the video's own
GPS again, each reads only the videos of its type. A video of another
type was reported as having no GPS: `process VIDEO --video_geotag_source
gopro` on a BlackVue video, which has GPS, failed it with "No GPS data
found from the video", and so did camm on a GoPro video.
Say instead "Not a BlackVue video: the geotag source blackvue reads
BlackVue videos only", and likewise for GoPro and CAMM, when the native
reader, limited to the videos of the type of the source's name, finds
none of that type in the video. The error stays the one that the native
reader reports, so that process() goes on with the video as before: the
next source tries it, and --filetypes leaves it out if it is of a type
that --filetypes does not name. The JSON form {"source": "gopro"} says
the same. The native source limited by its "filetypes", which is not
named after a type of video, reports as before.
Checked process on a folder of a GoPro, a CAMM, a BlackVue and an
unrecognized video and an image, with the gopro, camm, blackvue and
native video sources, exif with gopro, a native source limited to gopro,
and gopro then native, each without --filetypes and with gopro, camm,
blackvue, video, image, image,gopro and gopro,camm,blackvue (80 runs).
The entries and exit codes are those of before; only the messages of the
videos of other types than the named source change. video_process with
the gopro, blackvue and native video sources writes the same
descriptions, and the same sampling errors but for their messages.
Without a source path or a pattern, exiftool_xml reads the XML named after each image and video beside it in process. video_process gave it the video import path instead, as it gives every source without a path: the video was sampled, and its frames geotagged, by the XMLs in the video directory, matched by the video's path as written in them. An XML written in the video directory, `exiftool -X -ee -n clip.mp4 > clip.xml`, names the video by its name alone, and one copied from another computer by a path there, so `video_process DIR OUT --video_geotag_source exiftool_xml` failed such a video with "Cannot find the video in the ExifTool XML" (exit 7), while process found the XML, and so did video_process given the pattern "%g.xml" explicitly. In video_process, an exiftool_xml source without a source path or a pattern now reads the XML named after each video beside it, as in process, both to sample the video and to geotag its frames; for a video without one, it reads the XMLs in the video import path as before. To sample a video, given both a source path and a pattern, exiftool_xml reads the XML that the pattern names, where it is, rather than those at the source path, as it does for the frames of a video. A source path given with --geotag_source_path, or in the source, is read as before. Checked video_process on folders holding a BlackVue video with its XML named after it (made elsewhere, or in the folder), a video with only an XML of all videos with their full paths, a video with no XML, and two videos with one of each, and on one video named alone, with --geotag_source and --video_geotag_source exiftool_xml, the explicit "%g.xml" pattern, a --geotag_source_path, no source, and interval sampling. The videos with an XML named after them are sampled and geotagged as with the explicit pattern; the others as before. process of the same folders, with exiftool_xml alone or with --geotag_source_path, does not change; process of a video given both a pattern and --geotag_source_path now reads the XML that the pattern names where it is. The default video_process, process and sample_video do not change.
Distance samples of a video's own track are stamped by its straight GPS clock, and were clamped to the first and last GPS timestamps of the track. Those timestamps jitter about the clock, by up to 1.7 s on a GoPro, so a track whose last timestamp is early held every frame after it at that timestamp, up to the last point: on a trimmed GoPro MAX clip sampled every metre, the last 15 frames were all stamped 11:54:58.984, and process then spread them 1 ms apart over 8 m. A late first timestamp held the first frames the same way. Clamp the stamps to the clock's times at the first and last points of the track instead, so that frames are stamped by the clock up to the last point, and their times strictly increase. As before, a frame up to 1 ms of video past either end of the track, which a timelapse stretches to several ms of GPS time, is stamped at the end of the track, where the geotagging stage still finds it. The samples of a GPX or NMEA track keep the timestamps of the track, clamped to its first and last timestamps. On 49 sample videos sampled every 3 m, 10 change the stamps of their first or last frames only, by up to 2.1 s (17 frames of the trimmed clip, whose stamps now step at least 33 ms apart), and no position or other frame changes. The legacy gopro source now places the last frame of a GoPro MAX timelapse at the position the frame carries, 0.12 m from where it placed it before.
The README geotags a video stitched by Insta360 Studio by the original it was stitched from, through an exiftool_runtime pattern. The track read from the original is placed from the start of the stitched video, as a GPX track without a clock to sync it by is assumed to start with the video. Of a timelapse it lasts far longer than the video: an Insta360 ONE X original holds 110 seconds of GPS for a stitched timelapse of 1.77 seconds. process geotagged such a video with exit 0 and no word, where it warns of a GPX track that lasts more than twice as long as the video assumed to start with it (mapillary#861). Run the same check on a track read from a file that the pattern names other than the video, and warn the same way: "The GPS track in ... lasts 1.8 minutes and the video ... only 1.77 seconds, so the video could start anywhere along the track, and is assumed to start with it. Unless the video is a timelapse of the whole track, check that the exiftool_runtime pattern names the file that the video was made from". Only warn, as for a GPX track, since the track of a timelapse runs on past its end. The GPX warning is the one it was. Checked on Insta360 ONE X, X3 and X5 videos with their originals: the ONE X timelapse and an X5 video, whose original holds 13 minutes of GPS for 1.3 minutes of stitched video, warn, in process and video_process; X3 videos and timelapses whose originals match them do not. The descriptions do not change. Nor do the warnings and descriptions of exiftool_runtime without a pattern or with %f, on videos, images and a folder of both, nor those of GPX tracks.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stacked on #907; review the last commit.
The README geotags a video stitched by Insta360 Studio by the original it
was stitched from, through an exiftool_runtime pattern. The track read
from the original is placed from the start of the stitched video, as a
GPX track without a clock to sync it by is assumed to start with the
video. Of a timelapse it lasts far longer than the video: an Insta360
ONE X original holds 110 seconds of GPS for a stitched timelapse of 1.77
seconds. process geotagged such a video with exit 0 and no word, where
it warns of a GPX track that lasts more than twice as long as the video
assumed to start with it (#861).
Run the same check on a track read from a file that the pattern names
other than the video, and warn the same way: "The GPS track in ... lasts
1.8 minutes and the video ... only 1.77 seconds, so the video could
start anywhere along the track, and is assumed to start with it. Unless
the video is a timelapse of the whole track, check that the
exiftool_runtime pattern names the file that the video was made from".
Only warn, as for a GPX track, since the track of a timelapse runs on
past its end. The GPX warning is the one it was.
Checked on Insta360 ONE X, X3 and X5 videos with their originals: the
ONE X timelapse and an X5 video, whose original holds 13 minutes of GPS
for 1.3 minutes of stitched video, warn, in process and video_process;
X3 videos and timelapses whose originals match them do not. The
descriptions do not change. Nor do the warnings and descriptions of
exiftool_runtime without a pattern or with %f, on videos, images and a
folder of both, nor those of GPX tracks.
Unit and integration tests, doctests,
mypy --python-version 3.9,ruff check,ruff format --checkandusort diffare clean.