Difference between revisions of "AV1 Codec Testing"

Latest revision as of 11:56, 13 March 2020

Last updated 3/11/2020

AV1 is the newest emerging video coding format currently under development by AOMedia. I was interested in trying it out and seeing how it compares to my current favorite x265, and also the older x264. Keep in mind I'm doing this for my own purposes, so the following benchmarks are not comprehensive and are limited to my own intended use cases.

The AV1 library libaom-av1 is not currently available in ffmpeg by default. I found a good script someone made for compiling ffmpeg with support for AV1: https://gist.github.com/sparrc/026ed9958502072dda749ba4e5879ee3

At the time of this writing, the latest snapshot of libaom-av1 is:
1.0.0-errata1-avif-390-gc43904e50

Direct source from AOMedia:
https://aomedia.googlesource.com/aom/+/refs/tags/v1.0.0-errata1-avif

I downloaded the script:

# URL is very long and may wrap on multiple lines here.
wget https://gist.githubusercontent.com/sparrc/026ed9958502072dda749ba4e5879ee3/raw/e22698ead1984cd86b943f3473bd4bfb98591808/install-ffmpeg.sh

Note: As always, never download and run a script without reading it yourself to determine that it does not do anything malicious.

I ran the script:

sudo bash install-ffmpeg.sh

this script installs the custom ffmpeg into ~/bin/ffmpeg.

I chose a relatively small MTS file for testing:

du -h 00056.MTS

38M     00056.MTS

Runtime Benchmarks (Interlacing)

In the following benchmarks, the relevant time is the "real" time. The original MTS file is encoded with video:h264, audio:ac3, and has interlacing. Each encoding process I do below uses all defaults for the format, and encodes the audio to aac.

X264 Benchmark

time ~/bin/ffmpeg -loglevel -8 -i 00056.MTS -c:v libx264 out_x264.mp4

real    0m28.159s
user    1m43.388s
sys     0m0.176s

X265 Benchmark

time ~/bin/ffmpeg -loglevel -8 -i 00056.MTS -c:v libx265 out_x265.mp4

real    0m43.420s
user    2m40.852s
sys     0m0.304s

AV1 Benchmark

time ~/bin/ffmpeg -loglevel -8 -i 00056.MTS -c:v libaom-av1 -strict -2 out_av1.mp4

real    177m27.737s
user    413m16.620s
sys     0m6.688s

Note: The "-strict -2" parameter allows for the use of experimental codecs like AV1.

Conclusion

As you can see, the AV1 encoding took orders of magnitude longer.

• x264 = 28 seconds
• x265 = 43 seconds
• AV1 = 2 hours, 57 minutes, 27 seconds

Runtime Benchmarks (De-Interlacing)

In the following benchmarks, I decided to include de-interlacing.

X264 Benchmark (De-Interlacing)

time ~/bin/ffmpeg -loglevel -8 -i 00056.MTS -vf yadif -c:v libx264 outd_x264.mp4

real    0m29.251s
user    1m47.480s
sys     0m0.260s

X265 Benchmark (De-Interlacing)

time ~/bin/ffmpeg -loglevel -8 -i 00056.MTS -vf yadif -c:v libx265 outd_x265.mp4

real    0m42.207s
user    2m34.120s
sys     0m0.328s

AV1 Benchmark (De-Interlacing)

time ~/bin/ffmpeg -loglevel -8 -i 00056.MTS -vf yadif -c:v libaom-av1 -strict -2 outd_av1.mp4

real    162m54.203s
user    377m0.452s
sys     0m4.940s

Conclusion

The times are not significantly different from my first test that included interlacing, but interestingly choosing to de-interlace notably reduced the runtime for AV1.

• x264 = 29 seconds
• x265 = 42 seconds
• AV1 = 2 hours, 42 minutes, 54 seconds

File Size Comparison

The files starting with out_ are interlaced, and outd_ are de-interlaced.

du -h out_*;du -h outd*

11M     out_av1.mp4
15M     out_x264.mp4
5.6M    out_x265.mp4
8.2M    outd_av1.mp4
13M     outd_x264.mp4
4.1M    outd_x265.mp4

Conclusion

The best codec for file size, by a pretty large margin, is still x265. De-interlacing also resulted in smaller file sizes for all codecs.

Video Quality Comparison

In this test, I will take a cropped portion of the same frame from each file so that I can visually compare them.

I used ffmpeg to extract the 10th frame from each file. I chose the 10th frame because it had some motion that made the interlacing very visible.

for x in *.{MTS,mp4};do ffmpeg -i $x -vf "select=eq(n\,9)" -vframes 1 $x.png;done

Note: the "select" parameter takes a sequence starting at 0, so the 10th frame is 9.

I thought it was notable to look at the difference in file size of the images:

du -h *.png

2.9M    00056.MTS.png
2.1M    out_av1.mp4.png
2.0M    outd_av1.mp4.png
2.4M    outd_x264.mp4.png
2.1M    outd_x265.mp4.png
2.4M    out_x264.mp4.png
2.2M    out_x265.mp4.png

I then cropped a portion of each image starting at pixels x=0,y=250 with a size of 500x300. To do this automatically, I used the Imagemagick convert command:

for x in *.png;do convert $x -crop 500x300+0+250 ${x/\.png/_crop\.png};done

Image Comparison

Here I compare the frame sample for each codec. In case you're wondering, this is a video of me melting bismuth on the stove.

Conclusion

Obviously, interlaced video looks worse than de-interlaced across the board. Comparing de-interlaced x265 to AV1, I cannot tell the difference. Comparing de-interlaced x264 to x265, I'd say x264 looks a little sharper but the compression artifacts are a little more prominent. Overall I might prefer x264 for image quality, but x265 offers way better compression. AV1 seems to be worse for all my use cases so far, but it is still under development and maybe it will improve.

Update: I also tried different containers besides mp4, such as mkv, but I got identical results in all measurements, except that mkv gave slightly smaller file sizes for AV1.

Hardware Info

I performed these benchmarks on this machine:

lscpu

Architecture:        x86_64
CPU op-mode(s):      32-bit, 64-bit
Byte Order:          Little Endian
CPU(s):              4
On-line CPU(s) list: 0-3
Thread(s) per core:  1
Core(s) per socket:  4
Socket(s):           1
NUMA node(s):        1
Vendor ID:           GenuineIntel
CPU family:          6
Model:               42
Model name:          Intel(R) Core(TM) i5-2500K CPU @ 3.30GHz
Stepping:            7
CPU MHz:             1604.496
CPU max MHz:         3700.0000
CPU min MHz:         1600.0000
BogoMIPS:            6600.19
Virtualization:      VT-x
L1d cache:           32K
L1i cache:           32K
L2 cache:            256K
L3 cache:            6144K
NUMA node0 CPU(s):   0-3

sudo lshw -short -C memory|grep MHz

/0/f/0    memory         8GiB DIMM DDR3 Synchronous 1867 MHz (0.5 ns)
/0/f/1    memory         8GiB DIMM DDR3 Synchronous 1867 MHz (0.5 ns)
/0/f/2    memory         8GiB DIMM DDR3 Synchronous 1867 MHz (0.5 ns)
/0/f/3    memory         8GiB DIMM DDR3 Synchronous 1867 MHz (0.5 ns)