Mind Dump, Tech And Life Blog
written by Ivan Alenko
published under license Attribution-ShareAlike 4.0 International (CC BY-SA 4.0)copy! share!
posted in category Creativity / Video
posted at 21. Jun '18
last updated at 24. Jun '18

Video in Destiny Project

As of Jun, 2018

The goal is to serve videos to as many users as it is possible while using lowcost shared server.

<video> tag supports multiple versions of the video, so we can provide suitable formats to various clients. This requires much more space on the server side, because transcoding cannot be done on-the-fly and must be done beforehand on a beefy computer.

We should provide 3 versions at maximum, because of space and upkeep.

Types of users

  • Android - Google, released and prefers VP8, VP9
  • iOS/macOS - generally Apple is against patent free codecs, loves h264 and recently h265
  • totally free software operating systems (do not have patented technologies)
  • Windows - not sure if MS pays royalties for h264, software codecs (like x264) are used usually. Graphic cards from Intel, AMD and nVidia and some Intel processors have hardware decoding capability for h264 and VP9. VP9 support is recent one, Intel Kaby Lake (8xxx), nVidia Pascal (GTX1xxx), Polaris (RX 5xx).
  • Linux - depends on distro, on Ubuntu and distros with proprietary drivers, hw decoding is used. But majority of users have beefy computers, so no problems with any codec on fullHD. But vocal minority of users has typewriters with Pentium 2 or Core 2 Celeron and these guys have no hardware acceleration and cannot handle h264/720p and more
  • BSDs - no idea, assume the worst and software decoding is used, so similar to Linux boxes
  • exotic devices - depend on software decoding only
  • outside of <video> world - software of hardware decoding, depends on operating system, but not type of browser. Additional codecs can be installed, so performance is the main concern.

Types of codecs

video codec, audio codec, container, desired resolution/framerate/color space/number of colors
1. Ideal patent free: VP9 CRF 23 + Opus/FLAC + webm/matroska + 1080p+720p/30+60/YUV 4:2:0/8-bit
2. Accelerated fallback: h264 + AAC + MP4 + 1080p+720p/30+60/YUV 4:2:0/8-bit
3. Fallback patent free + very slow and old machines: VP8 + Vorbis + webm + 720p,VGA,800x600/30/4:2:0/8-bit

Dream/master copies: VP9 + Opus + webm + 1080p,4k/4:4:4/10-bit (for download)
Future ideal: AV-1 + Opus/FLAC, should be ready for Tokyo Olympics in 2020….I hope

Notes:

  • Chroma subsampling - more than 4:2:0 (i.e. 4:2:2 and 4:4:4) are not hardware decoded like never
  • Colors - generally 8-bit are hardware decoded, 10-bit not
  • Audio: https://wiki.xiph.org/Opus_Recommended_Settings
  • nVida and ffmpeg: https://developer.nvidia.com/ffmpeg
  • Hardware acceleration and hardware decoding are distinct terms (accelerated on GPU/decoded by dedicated silicon), but for our purposes it’s the same
  • h265/HEVC - while it is supported somewhere, VLC/mpv, Apple recently started to support it, but the situation is not certain and some patent holders don’t know how much they want to ask to pay and terms. I’d be dumb to use this codec
  • 10-Bit H.264 explanation: https://gist.github.com/l4n9th4n9/4459997
  • also 10-bit support is good in Linux, BSD and mobile, but very awkward in Windows (thanks nVidia and AMD!). Which is fucking ridiculous, because 4k televisions can have 10-bit colors (some with FRC, but whatever), so they should allow 10-bits on gaming graphics cards, not only on professional ones

Android

source: https://developer.android.com/guide/topics/media/media-formats

VP9 and h264 main profile looks the best. VP8 should handle Android 5 and less.

Windows/IE+Edge, Chrome, Firefox, Safari

https://developer.mozilla.org/en-US/docs/Web/HTML/Supported_media_formats https://support.apple.com/en-us/ht207022

VP8+Vorbis is supported everywhere. But quality and file size is awkward. So this needs balancing.
VP9+Opus in Firefox, Chrome and new Opera.
HEIF/H265 in macOS High Sierra and iOS 11….yeah, whatever. Other than what h264+AAC is a safe way to deal with these devices.

Thumbnails

Thumbnailer should generate N preview images and shouldn’t depend on the length of the video. Then classic view should show it as a image gallery and “modern” view should play it in a thumbnail of related video or in search when it’s hovered (or something similar on mobile) like on YouTube or Pornhub.

    Extract one picture each 50 frames:

    thumbnail=50

    Complete example of a thumbnail creation with ffmpeg:

    ffmpeg -i in.avi -vf thumbnail,scale=300:200 -frames:v 1 out.png

Webm = Web Media

The same as Matroska, but allows only VP8/VP9+Vorbis/Opus+VTT. Also it’s a good idea to have only one video and one audio track per video, with N subtitles just like YouTube has.

Script and sources

  • http://wiki.webmproject.org/ffmpeg/vp9-encoding-guide <– this is so fucking old, don’t trust it
  • https://ffmpeg.org/ffmpeg-all.html
  • https://trac.ffmpeg.org/wiki/Encode/VP9
  • https://developers.google.com/media/vp9/settings/vod/ <– this is awesome
  • https://wiki.xiph.org/Opus_Recommended_Settings
  • https://multimedia.cx/eggs/supplying-ffmpeg-with-metadata/
#!/bin/sh
# we assume the input video is in fullHD/30fps
# TODO license and URL of origin
TITLE="TODO: test title"
DESCRIPTION="aaa"
COMMENT="Generated by Destiny Project"
SOURCE=$1
TARGET=$SOURCE
THREADS=16

# -profile baseline/main -preset slow -tune film/animation/stillimage
echo "creating h264 version..."
ffmpeg -i $SOURCE -vf scale=1920x1080 \
  -threads $THREADS \
  -crf 23 -c:v libx264 -c:a aac -b:a 128k \
  -profile:v main -preset slow \
  -pix_fmt yuv420p -r 30 \
  -metadata title="$TITLE" \
  -metadata description="$DESCRIPTION" \
  -metadata comment="generated by Destiny Project" \
  -y ${TARGET}-x264main.mp4

# profile doesn't work in Debian testing 2018/06 ffmpeg
echo "creating h265 version..."
ffmpeg -i $SOURCE -vf scale=1920x1080 \
  -threads $THREADS \
  -crf 28 -c:v libx265 -c:a aac -b:a 128k \
  -preset slow \
  -pix_fmt yuv420p -r 30 \
  -metadata title="$TITLE" \
  -metadata description="$DESCRIPTION" \
  -metadata comment="generated by Destiny Project" \
  -y ${TARGET}-x265.mp4

echo "creating VP9 version..."
# from https://developers.google.com/media/vp9/settings/vod/ + set pixfmt and fps
ffmpeg -i $SOURCE -vf scale=1920x1080 -b:v 1800k \
  -minrate 900k -maxrate 2610k -tile-columns 2 -g 240 -threads $THREADS \
  -quality good -crf 31 -c:v libvpx-vp9 -c:a libopus \
  -pass 1 -speed 4 -pix_fmt yuv420p -r 30 $TARGET-24-30fps.webm && \
ffmpeg -i $SOURCE -vf scale=1920x1080 -b:v 1800k \
  -minrate 900k -maxrate 2610k -tile-columns 4 -g 240 -threads $THREADS \
  -quality good -crf 31 -c:v libvpx-vp9 -c:a libopus \
  -pass 2 -speed 4 -pix_fmt yuv420p -r 30 \
  -metadata title="$TITLE" \
  -metadata description="$DESCRIPTION || generated by Destiny Project" \
  -y $TARGET-vp9.webm


echo "creating VP8 version..."
ffmpeg -i $SOURCE -vf scale=hd480 -b:v 1800k \
  -minrate 900k -maxrate 2610k -g 240 -threads $THREADS \
  -crf 10 -c:v libvpx -c:a libvorbis \
  -pix_fmt yuv420p -r 30 \
  -metadata title="$TITLE" \
  -metadata description="$DESCRIPTION || generated by Destiny Project" \
  -y $TARGET-vp8.webm

Conclusion and TODO

  • use the script
  • metadata extractor?
  • test baseline and main profiles same CRF for file size (h264)
  • can I mix h264 main and baseline profile and lowend device cannot play main, will it use baseline?
  • this was fun to research, but I’m exhausted :-D

Add Comment