FrameTimecode.frame_num is inaccurate for VFR video
FrameTimecode.frame_num is currently approximated from the timestamp and average frame rate for VFR video. This can produce incorrect frame numbers for consumers which require an actual frame ordinal.
This came up in #565, but also affects existing functionality such as save-qp and ThresholdDetector, so it should probably be handled separately.
Expected behavior
For frames read from a video, frame_num could represent the zero-based ordinal of the frame in the presentation sequence actually indexed/decoded by PySceneDetect:
frame_num = 0, 1, 2, 3, ...
PTS would remain the authoritative temporal coordinate.
VFR itself does not prevent assigning an exact frame ordinal; it only means that the time between consecutive frames is variable.
For corrupted media, this would not attempt to reconstruct frames which may have existed in an undamaged version of the file. If a corrupt frame is emitted, it is counted. If no frame is emitted, there is no frame in the processed sequence to count.
One caveat is that different backends may recover severely malformed media differently, so frame ordinals could differ between backends in those cases.
Possible approach
One option would be to store the exact frame ordinal along with PTS for frames as they are processed.
A compact frame-to-PTS index could be maintained as:
pts_by_frame[frame_num] = pts
This would allow:
- Exact frame arithmetic using
frame_num.
- Exact frame -> PTS lookup.
- PTS -> frame lookup using binary search.
- Time-based operations such as "the frame displayed 2 seconds ago" by calculating the target PTS and looking up the frame containing that time.
The memory overhead should be small if PTS values are stored as int64 (8 bytes/frame).
This is only one possible implementation and seeking/start-from-middle behavior would still need to be investigated, since the absolute ordinal cannot be determined from PTS alone if the preceding frames have not already been indexed.
Known affected areas
Related: #534, #565
FrameTimecode.frame_num is inaccurate for VFR video
FrameTimecode.frame_numis currently approximated from the timestamp and average frame rate for VFR video. This can produce incorrect frame numbers for consumers which require an actual frame ordinal.This came up in #565, but also affects existing functionality such as
save-qpandThresholdDetector, so it should probably be handled separately.Expected behavior
For frames read from a video,
frame_numcould represent the zero-based ordinal of the frame in the presentation sequence actually indexed/decoded by PySceneDetect:PTS would remain the authoritative temporal coordinate.
VFR itself does not prevent assigning an exact frame ordinal; it only means that the time between consecutive frames is variable.
For corrupted media, this would not attempt to reconstruct frames which may have existed in an undamaged version of the file. If a corrupt frame is emitted, it is counted. If no frame is emitted, there is no frame in the processed sequence to count.
One caveat is that different backends may recover severely malformed media differently, so frame ordinals could differ between backends in those cases.
Possible approach
One option would be to store the exact frame ordinal along with PTS for frames as they are processed.
A compact frame-to-PTS index could be maintained as:
This would allow:
frame_num.The memory overhead should be small if PTS values are stored as
int64(8 bytes/frame).This is only one possible implementation and seeking/start-from-middle behavior would still need to be investigated, since the absolute ordinal cannot be determined from PTS alone if the preceding frames have not already been indexed.
Known affected areas
FrameTimecode.frame_numsave-keyframes(Add keyframe format v1 CLI export #565)save-qpThresholdDetectorframe_numidentifies an actual frame in VFR videoRelated: #534, #565