Comparative analysis of uncompressed AVI and FFV1 video

Graham Purnell is a former photographer, web content and social media professional who began working as a Digital Preservation Assistant with the National Library of Scotland in early January 2019

The purpose of these tests: to use FFMPEG to transcode an uncompressed AVI file to FFV1, with identical ‘pixel-for-pixel’ output to the original source video, and to verify the results.

I’ve read a lot of information on the internet about the benefits of using FFV1/MKV as an archive video format/wrapper, but little empirical evidence about why it is suitable. It doesn’t help that, in the minds of many people, the word ‘compression’ is synonymous with ‘lossy’ (“if file sizes are smaller, surely some data must be missing.”) Archivists and digital preservation practitioners avoid much of the technical video information when writing, and video specialists address the subject in terms beyond the scope of many archivists. Thankfully, I have some previous video production experience and I hope I can help make some sense of it all.

Data doesn’t necessarily need to be removed from a video file to compress it. Much of an uncompressed file’s data stream may be zero padding, redundant information that can be removed because it holds no actual information. FFV1 also uses an efficient entropy coding system that decodes to the same raw video output as the original source; essentially, it’s a more efficient way of coding the same information. The codec name (coder/decoder ) is a good one, because what codecs do is synonymous with coding and decoding a cipher; different ciphers exist to encrypt information, some longer and more complex than others, but what really matters is that no information is lost on deciphering.  Please don’t ask me about FFV1’s entropy encoding of image information, it’s way beyond my ken, but it is in the FFV1 file specification, for those that are interested (https://www.ffmpeg.org/~michael/ffv1.html), and a concise explanation is also given on the FFV1 Wikipedia page (https://en.wikipedia.org/wiki/FFV1)

But how do we prove FFV1 is lossless? Conveniently, FFMPEG has a framemd5 function that allows us to run MD5 fixity checks on every single frame of a video. I transcoded a file from uncompressed AVI to FFV1 video plus FLAC audio in an MKV wrapper, and ran framemd5 checks for video and audio on both. The results can be seen in the linked spreadsheet, which has a colour coded explanation of what each tab means http://bit.ly/AVItoFFV1 (please download the file – it has conditional formatting that doesn’t display when viewed in a browser.)

The framemd5 output for video shows that the FFV1 transcoded file is frame-by-frame identical to the AVI source file. If you want to see how conditional formatting in the spreadsheet works, change a digit in one of the fields in the FFV1videoMD5 sheet and tab to the next field. You will see the amended data field change colour. That none of the md5 checksums in the FFV1videoMD5 sheet are coloured, shows they are identical to the checksums on the sheet it’s being compared to, i.e. AVIvideoMD5, the source video file information.

I also transcoded back to uncompressed AVI (v210 codec) from the FFV1 file and ran a framemd5 check to see if the reconverted AVI video output was identical to the source AVI. It was, and this is also included on the spreadsheet.

I chose to transcode audio to FLAC for research purposes, but wouldn’t recommend it as a video soundtrack format for preservation masters. Framemd5 decodes to raw video on-the-fly, from which it creates its checksums, but there is no standard ‘raw audio’ equivalent. By default, framemd5 pipes sound as 16 bit little endian PCM (CD quality audio) before creating checksums, which may not be the quality of the original sound. Different audio codecs can be forced in FFMPEG to match the original audio, but this is fairly complex and, since audio is (with the exception of subtitles or metadata) the smallest component of a video file, it is recommended to use the ‘-c:a copy’ command when transcoding; this means c(odec):a(udio) copy and creates a transcoded video file with identical audio to the source, eliminating the possibility of wrongly transcoding sound.

Something picked up by one of our video specialists (and the eagle eyed among you may spot) is that the FFV1 file is identified by MediaInfo as a ‘progressive’ file format (i.e. not interlaced.) This is literally true because two fields, each making up half the frame information in interlaced video, are combined (or interleaved) into a single frame in the FFV1 file, effectively making the file format progressive*. In interlaced video, odd lines and even lines are displayed consecutively, persistence of vision fooling the brain that a full frame has been displayed at once; this is why a 50Hz, 25 frames-per-second, interlaced video is, more literally, a 50Hz, 50 fields-per-second one. (Edit: the identification of the FFV1 transcode as 'progressive' possibly has more to do with the v210 codec** of the original source file than the ability of FFV1 to store interlacing information. See the Notes section on this page: https://www.loc.gov/preservation/digital/formats/fdd/fdd000353.shtml#notes)

Fortunately, FFMPEG has an interlace detection filter. You can read about it here: http://www.aktau.be/2013/09/22/detecting-interlaced-video-with-ffmpeg/

By running the ‘idet’ filter on the source and transcoded files, I was able to demonstrate that, when output as raw video, the FFV1 output removes the ‘progressive’ flag, correctly delivering the content as interlaced video, identical to the original. The idet filter only needs to be run on part of the video to detect interlacing, making a check very quick; I chose 1800 frames. Interlacing of this FFV1 file will, of course, be disregarded on modern, progressive displays but will display correctly on older, interlaced displays.

The FFMPEG ‘idet’ output, from source and transcoded video files, is in this text file: http://bit.ly/interlace_check - this file also has links to the page above, describing ‘idet’ & what its FFMPEG output means, and the FFV1 interlacing ‘bug report’ (that wasn’t really a bug.)

In the ‘idet’ information, you may notice that the FFV1 transcode INPUT video file is identified as yuv422p10le(progressive) i.e. ‘YUV 4:2:2 planar 10bit little endian (progressive)’. In the ‘idet’ raw video OUTPUT, it has been identified as yuv422p10le and the ‘progressive’ flag is no longer there. The interlacing detection output is at the bottom of each test (source and transcode) and is identical for both files (apart from the ‘Parsed_idet_0’ unique file numbers):

[Parsed_idet_0 @ 0602cb80] Repeated Fields: Neither:  1799 Top:     0 Bottom:     2

[Parsed_idet_0 @ 0602cb80] Single frame detection: TFF:  1076 BFF:    26 Progressive:    11 Undetermined:   688

[Parsed_idet_0 @ 0602cb80] Multi frame detection: TFF:  1800 BFF:     0 Progressive:     0 Undetermined:     1

High TFF or BFF figures (Top Field First or Bottom Field First) show that the video is interlaced. The idet filter tries to determine interlacing for individual frames first, then for multiple frames. False positives in the single frame detection have been ‘weeded out’ by multi-frame detection.

In summary, FFV1 transcoded files losslessly retain all video information and have been proven to retain interlacing. I think these tests show that FFV1 is truly lossless and can confidently be used as an archival preservation master format.

I hope my blog post has been useful, hasn’t confused anyone, and if you have any further questions, please email me at g.purnell@nls.uk

* This isn't true of all FFV1 files, but is true of the v210 to FFV1 transcoded file under test. I also didn't force ffmpeg to interlace the transcoded file, because I wanted to simplify the command and see how many of the original file's attributes would be carried over automatically.

** Peter B. (Twitter handle @pjotrek_b) used an identical FFMPEG command to convert a v210 MOV file to FFV1 and MediaInfo recognised it as interlaced. This suggests that the misreporting of my FFV1 file as 'progressive' has more to do with the AVI container than the v210 encoding.